worked on my most interesting data engineering project yet. a modular Python ETL/ELT pipeline that ingests weather forecast data from a public REST API into a star schema db, with Airflow orchestration, structured logging, data validation, and pytest unit tests.
If you’re early in your career maybe it’s helpful to hear that despite several decades leading large eng projects I still don’t really know what I’m doing. Every new project I’m still filled with that sense of “holy crap how do I solve this?” You always get better but if you’re doing your best work you’ll probably always be operating outside your comfort zone. Don’t be fooled into thinking we’ve all got it figured out, just figure out the smallest next step you can take towards solving that problem you thought was too hard.