Hi everyone,
I have around 13 years of experience in IT, primarily in data engineering, but I feel quite stuck in my career and would appreciate some honest advice from people who have been in a similar situation.
**A bit about my career journey:**
My first four years in India were quite unstructured. I was moved between different projects and cities, mostly doing low-level development work. I was trying to adjust to new environments and projects, so I don't feel I built a particularly strong technical foundation during those years.
After moving to the UK, I joined my second company and got an opportunity to work with big data technologies. I spent around four years working with Hive, Impala and some Scala, and got introduced to Spark. I learned on the job, but I never became particularly proficient in Scala or Spark.
I then moved to another company, where I picked up some PySpark in a much faster-paced environment. Unfortunately, I was made redundant after around 1.5 years. I also had a performance review during that period, which made me reflect on the gaps in my technical skills.
I am currently in my third company in the UK. The codebase here is old, complex and difficult to understand. During my first year, I was learning the existing code while also working on a migration project. Now that the migration is complete, most of my work involves BAU support, troubleshooting and occasional enhancements.
The team also has other developers who handle much of the new development. Over time, I have realised that coding has never really been my strongest skill. I have tried learning Python and Scala from scratch several times, but I struggle to stay consistent and get to a level where I feel genuinely confident. I can work with SQL and understand data engineering concepts, and I have experience with technologies such as Python, PySpark, Scala and AWS, but my practical coding skills are not as strong as my CV might suggest.
I have even received interview opportunities from companies like Meta twice in the past two years, but I haven't been able to clear the initial coding rounds. I did prepare, but I struggle to solve problems quickly and confidently under interview conditions.
**Where I am struggling now:**
With 13 years of experience, I feel I should be operating at a senior engineer or tech lead level. Instead, I often lack confidence in my technical abilities and worry that people will eventually realise I am not contributing as much as someone with my experience should be.
In my current team, another colleague works closely with the developers, translates business requirements and validates the results. In practice, he takes on much of the coordination and technical leadership, while I handle BAU work and some other tasks. I have proactively asked my peers for more responsibility and have been given some architectural and design-related work, which I am trying to develop further. However, I am still learning and don't yet feel that I am operating at an architecture or tech lead level.
My concern is that the team is gradually shrinking, and I am not sure how to position myself for the future.
**I see a few possible paths, but I am confused about which one makes sense:**
- **Continue in data engineering:** Improve my Python/PySpark skills and work towards becoming a senior or lead data engineer. However, I am honestly tired of repeatedly trying to learn programming and feel reluctant to spend another few years focusing heavily on coding.
- **Move towards data architecture or solution design:** I enjoy the idea of understanding systems, designing solutions, documenting architecture and working with stakeholders. I have started getting some exposure to this, but I don't know how to bridge the gap from my current role to a genuine architecture position.
- **Move into project management or product management:** I wonder whether a role with less hands-on coding might suit me better. However, I don't know whether this would be a sensible transition after 13 years in a technical career, or whether I am simply considering it because I have lost confidence in coding.
There is also the constant pressure to keep up with new technologies such as Databricks, Snowflake, dbt and AWS. I feel I need to learn all of these to remain employable, but I am already struggling to decide where to focus my energy.
I also know data engineering is a good paying career so I don’t know if I am making a right decision if I leave it. But when I go for job hunt, in most of the interviews I failed at first stage which is coding stage.
I don't necessarily expect to become an expert in every technology. I just want to develop a clear area of strength, become confident in my work and feel that I am making a meaningful contribution to my team.
**For those who have been in a similar position, I would really appreciate your advice:**
- Have you ever reached a point in your career where you realised that hands-on coding wasn't your strength? What did you do next?
- Is it realistic to move into data architecture or solution design without being an exceptionally strong programmer?
- Would moving into project management or product management make sense, or would I be starting from scratch in another career?
- If you were in my position, would you invest another year in improving your technical skills or start planning a transition into a different role?
I would especially appreciate advice from people who have made a similar transition after 10–15 years in IT. I am looking for practical, honest advice rather than simply being told to learn more technologies.
Thanks in advance!