PgAI – LLMs have been part of everyday life already for a while. One aspect I think has not been explored well so far is using them as part of ETL. The implementations I have seen so far don’t take advantage of batch APIs and are not standardized to enable the easy replacement of a model. Having said that, I believe those hurdles will be overcome soon.
https://github.com/timescale/pgai
Related links
- Snowflake cortex complete – call llm functions from snowflake https://docs.snowflake.com/en/developer-guide/snowpark-ml/reference/1.5.0/api/model/snowflake.cortex.Complete)
- OpenAI Postgres extension – allows calling OpenAI or local models using Ollama. https://www.crunchydata.com/blog/accessing-large-language-models-from-postgresql
- Do you use LLMs in your ETL pipelines – Reddit discussion regarding using LLMs in ETL pipelines. There are pros and cons of course, specially for cases you need a deterministic answer but there are also other cases –
https://www.reddit.com/r/dataengineering/comments/1h17gjm/do_you_use_llms_in_your_etl_pipelines/
Life Altering PostgreSql Patterns – a back-to-basics post. I agree with most of the points mentioned there, specifically around adding creaetd_at, updated_at, and deleted_at attributes to all tables and saving state data as logs rather than saving only the latest state. I found the section about enum tables interesting. This is the first time I was exposed to this idea, and the ability to add a description or metadata is excellent.
https://mccue.dev/pages/3-11-25-life-altering-postgresql-patterns
Via this post, I learned about the on update cascade option, you can read more about it here – https://medium.com/geoblinktech/postgresql-foreign-keys-with-condition-on-update-cascade-330e1b25b6e5
AI interfaces of the future – I usually don’t share videos, but I think this talk is thought-provoking for several reasons –
- Gen UI patterns – an emerging field, the talk reviews several products and highlights good and destructive patterns. Some of the patterns, like suggestions or auto-complete, are transparent to us but are present in many products we know, and that’s something important to notice when you build such a product.
- Product review: Knowing what is out there is good for inspiration, ideas, and understanding the competitive landscape. However, new products are coming out every day, and it is hard to track all of them.
Simplify Your Tech Stack: Use PostgreSQL for Everything – Two widespread tensions, especially in startups, are build vs. buy conflicts and using specialized products or technologies (e.g., different databases) that are top of the breed but not many people can use and maintain vs. more common technology that more people can maintain but can have performance drawbacks or other limitations. Mainly working in startups, I usually prefer to use standard technology to run faster, knowing that the product, focus, and priorities often change. With that being said, I acknowledge that early adoption of new technologies can be life-changing for a startup, but figuring out what to bet on is hard.
https://medium.com/timescale/simplify-your-tech-stack-use-postgresql-for-everything-f77c96026595
CDK Monitoring Constructs – if you are using AWS CDK as your IAC tool, CDK monitoring constructs enable you to create cloudwatch alarms and dashboards almost out of the box. I wish they would release and add additional options at a faster pace.
Tom Ron 