Solutions Architect: Databricks & Big Data

Who Are We?

SDG Group is a global consulting powerhouse with over 30 years of experience as a Market Leader in Data & Analytics. As we continue to push the expansion of our US business, we are seeking a Solutions Architect to lead technical delivery and innovation for our most ambitious client engagements.

Why Join Us?

In this role, you will be the technical backbone of our Databricks offering. You will design sophisticated, large-scale Big Data architectures and actively engage with the Databricks organization in close collaboration with our Alliances Lead to showcase SDG’s premier technical capabilities. You are not just building pipelines; you are architecting the future of data intelligence for the world’s leading brands.

• Innovation-First Culture: We don't just use technology; we evaluate and adopt the most disruptive tools in the market before they become mainstream.

• Global Reach, Local Impact: Benefit from the resources of a 2,500+ person global firm while playing a pivotal role in our targeted US expansion.

• Continuous Evolution: Enjoy a meritocratic career path where your technical contributions are recognized and your growth is supported through official certifications and specialized training.

What You’ll Be Doing

• Advanced Big Data Architecture: Design and define end-to-end, high-performance architectures for processing massive, complex volumes of data in distributed environments. Establish standards for Data Lakehouse and Modern Data Warehouse patterns to ensure scalability and efficiency.

• Storage Strategy: Implement advanced storage strategies using Delta Lake, Apache Iceberg, and optimized formats like Parquet.

• Databricks Solution Design: Lead the technical design of solutions on the Databricks Data Intelligence Platform, utilizing Unity Catalog for unified governance.

• Future-Proofing: Demonstrate deep experience with Delta Live Tables (DLT) and maintain awareness of the upcoming Lakeflow ecosystem (Spark Declarative Pipelines) to ensure SDG remains at the cutting edge of the roadmap.

• Performance Tuning: Provide expert guidance on performance tuning for large Spark clusters and optimizing SQL workloads within Databricks.

• Client Engagement: Act as the "Technical North Star" in client workshops, translating strategic business needs into viable, high-impact technical blueprints.

• Alliance Collaboration: Collaborate with the Alliances Lead to engage with Databricks’ technical teams, demonstrating SDG’s excellence through deep-dive architectural reviews and Proofs of Concept (PoCs).

• Technical Excellence: Support project execution from a technical standpoint, ensuring the team adheres to DataOps best practices, CI/CD, and robust data quality frameworks.

What We Look For

• Architecture Experience: 6+ years of professional experience in data engineering, with a focus on building solutions on distributed platforms and managing complex ETL/ELT pipelines.

• Lifecycle Management: Proven experience in architecting solutions that manage the full data lifecycle—ingestion, cleaning, and governance—at enterprise scale.

• Databricks Expertise: 3+ years of hands-on experience designing and deploying solutions within the Databricks ecosystem.

• Cloud Platforms: Deep experience with at least one major cloud provider (Azure, AWS, or GCP), particularly their data-related services.

• Languages: Proficiency in Python, Spark, PySpark, and/or their variants such as SparkSQL.

Desired Skills & Experience

• Orchestration: Hands-on knowledge of orchestrators such as Airflow or native Databricks tools like Databricks Workflows.

• Processing Frameworks: Advanced experience with frameworks such as dbt.

• Communication: Exceptional interpersonal and communication skills, with a proven ability to lead technical discussions with both developers and executive stakeholders.

• Innovation: Collaborate with our global innovation labs to implement proprietary accelerators and artifacts that reduce time-to-market for our clients.

• Mindset: A proactive, "Growth Mindset" with a passion for staying at the forefront of emerging data technologies.

Benefits of Working With Us

At SDG, we’re constantly striving towards building a culture where people come first by creating an environment of transparency, collaboration, and meritocracy. We take care of our global employees by making sure we offer the following:

• Competitive Compensation: Full-time employment in an exciting and growing high-tech industry with a competitive salary.

• Career Growth: A professional career with opportunities for personal/professional growth and rapid advancement based on meritocracy.

• Feedback Loop: Regular meetings with your manager to share feedback, discuss new challenges, and review your career trajectory.

• Flexibility: A creative and innovative work environment with flexible working options (flexible working hours, good mix of on-site and remote).

• Health & Wellness: Robust Healthcare and Benefits package including Medical, Dental, Vision, Disability coverage, and various other benefit options.

• Retirement: 401k Retirement plan with Employer match.

• Time Off: 3+ weeks paid vacation and flexible holiday schedule.