New in Community Say hello to the Genie Hub Your questions about Databricks Genie, answered by the people actually building with it. Ask the community anything about Genie… Ask → Genie runs on plain-language conversation - you ask, it answers f...
Hey Community! Join us for a free Databricks Community sponsored BrickTalk on Thursday, July 30! BrickTalks is a community event series where Databricks experts share real-world use cases, demos, and practical insights for building with data and AI, ...
Community Ask Hey Community! Your Databricks Community profile is often the first thing other members see when they come across your posts, solutions, blogs, or event participation. A complete profile helps other members understand who you are,...
Help Your Knowledge Reach More People Every Technical Blog, Community Article, MVP article, or answer you share can help someone beyond the Databricks Community. Small improvements in how you write can make your content easier to discover through C...
WEBINARTime: July 21 / 8:00 AM PT Whether you’re building an agentic AI roadmap or pressure-testing your current approach, you’ll gain a clearer view of how leading enterprises are approaching agentic AI, and where the biggest opportunities lie ahead...
Monthly Recap • June 2026 Community Fellows Community Fellows are Bricksters who show up – in the forums, for customers, and for the community. Some months, that means going beyond the forums. June was one of them – Fellows brought their energy t...
Driving seamless access to data on social determinants of health is important for helping healthcare professionals better understand health inequities across social groups. The Social Determinants of Health Solution Accelerator shows how healthcare d...
The Weekly Digest • July 13 – July 19 Community Pulse Another week, another wave of knowledge. Here's what the community built... This Week's Community Stars Shoutout to this week's standout contributors – and a special welcome to a first-time...
General Q: Where can I ask questions if I have questions while going through the self-paced material? A: The Databricks Academy Learners Group page is a forum within the Databricks Community where you can ask questions and get them answered by your ...
Dear Team,I have completed all the required courses within the specified time period. However, I have still not received the exam voucher.Could you please check the status of my voucher and let me know if any additional action is required from my sid...
New in Community Say hello to the Genie Hub Your questions about Databricks Genie, answered by the people actually building with it. Ask the community anything about Genie… Ask → Genie runs on plain-language conversation - you ask, it answers f...
Really happy to see the Genie Hub launch, Tushar and team.This is a great place for people to ask questions, share what they are building, and learn from others who are already working with Genie in real projects.Today, Genie helps people ask questio...
Dear Community Experts,I need your expert advice and suggestions on development of data quality framework. What are powerfull data quality tools or libraries are good to go for development of data quality framework in Databricks ? Please guide team.R...
There are plenty of tools available but one that's closely integrated with Databricks is DQX. It is quite an awesome tool that really covers most of the rules you would need. It also has the ability to develop complex SQL queries wherein joins and ag...
I'm working with Delta tables using the Iceberg Uniform feature to enable Iceberg-compatible reads. I’m trying to understand how metadata cleanup works in this setup.Specifically, does the VACUUM operation—which removes old Delta Lake metadata based ...
Sound discussion overall in this thread. Two hard earned lessons that came out of running this in production that I can think of sharing:(1) Rehearse the cleanup on a scratch table before relying on it: point your reader at a test table, expire snaps...
Hi All, I need an up-to-date plain Delta copy of a streaming table created by Lakeflow pipeline. My first thought was CLONE, and that path is closed. The clone docs say streaming tables and materialized views cannot be the source or target of a deep ...
Thanks @adnan_alvee , yes, we did test Compatibility Mode. The output is a copy of files at a storage location, not a Unity Catalog table.Context: We are setting up an external engine (duckdb) to read UC Delta tables through credential vending, which...
As data engineers, we spend a significant amount of time writing SQL queries to ingest, transform, and analyse data. However, writing a query that returns the correct results is only part of the equation. Equally important is understanding how Spark ...
hi,Trying to set up serverless compute and can't create a private endpoint because "Private access endpoint service {endpoint name} is not supported because {endpoint name} is not supported for account {account name}". Following this thread but stuck...
On AWS, all private connectivity from serverless compute runs through a network connectivity configuration. The NCC is an account-level object that holds private endpoint rules, and attaching it to a workspace is what routes serverless traffic throug...
It looks like we have another workspace-side notebook UI regression for R plots (something similar happened last May).This time, R plots are displaying but have several very strange symptoms related to scaling:1. Larger figures (that are too big to b...
If we set the REPLICA option in Postgres, we can then set CDF sync to Unity Catalog in the branch overview. #databricks more news and video https://medium.com/databrickscommunity/databricks-news-rt-lakehouse-reyden-lakebase-ttl-8416bdccf627
Is there a way to setup CDF through the DAB instead of doing these UI steps?
Hi there, I have what should be a somewhat straightforward implementation of a Custom Community Connector. I've tried to Dry run a few times but am encountering PYTHON.MODULE_NOT_FOUND_ERRORBelow is some context about my setup:Source Relative path: l...
the sys.path approach won’t fix this for Lakeflow Connect pipelines. During a dry run, Databricks imports databricks.labs.community_connector before your ingest notebook runs, using only the pipeline’s Environment dependencies. Fix: Pipeline Settings...
Hi there, I'm a bit confused about how connections for Community Connectors setup should be. I assumed that the `connector_spec.yaml` file should outline the specifics of what is required for a Community Connector, but in practice it doesn't seem lik...
The connector_spec.yaml isn't what controls the UI for custom connectors. When you use the "Add Custom Community Connector" path in the workspace UI, Databricks shows a generic connection form with all available auth types. That form is the same for ...
Environment:Cloud: AzureDatabricks Account Console (accounts.azuredatabricks.net)Access Connector: system-assigned managed identity (no user-assigned identity involved)Region: East USIssue:Creating a Unity Catalog storage credential via the Account C...
Yes, this is a bug in the Account Console UI. You've already diagnosed it correctly: for a system-assigned managed identity, managed_identity_id should be omitted entirely from the request payload. An empty string "" is not the same as omitting the f...
Hello,I created a new Databricks account through the AWS Marketplace $400 trial using my business email. The workspace is in us-east-1, the AWS Marketplace payment method is connected, and the full $400 credit balance is active.Every Databricks-hoste...
That 403 with "rate limit of 0" isn't something you accidentally configured, it's a Databricks-imposed gate that applies to newly created accounts. Even with active trial credit and a valid payment method on file, Foundation Model API access gets hel...
Hi there, I'm trying to run a pretty barebones test here with community connectors. I'm trying out the "vanilla" Microsoft Teams connector by:(A) going to the Data Ingestion tab in our workspace,(B) selecting the Microsoft Teams community connector,(...
Both issues trace back to running on non-serverless compute. Community connectors were designed to run on serverless Spark Declarative Pipelines (SDP), and serverless is on by default in the CLI config for a reason. The No module named 'databricks.la...
Materialized View backing pipeline retains old Unity Catalog after catalog rename (dev_cat → dap_catalog_dev) resulting in NO_SUCH_CATALOG_EXCEPTIONThis issue blocks the creation and refresh of Materialized Views after a Unity Catalog rename.Workspac...
What you're hitting is expected behavior, not something you did wrong. Every materialized view is backed by a managed serverless pipeline, and that pipeline pins its catalog, target schema, and internal storage location at the moment the M...
| User | Count |
|---|---|
| 2280 | |
| 1356 | |
| 1241 | |
| 1184 | |
| 961 |
| User | Count |
|---|---|
| 445 | |
| 286 | |
| 220 | |
| 63 | |
| 51 |
Most teams I work with don't struggle with the AI part of MLOps. They struggle with the boring parts: getting code from a notebook into production without breaking it, knowing when a model has gone st...
Build a sub-second operational dashboard on live streaming data, starting with a live Kafka stream This blog is a full end-to-end walkthrough for practitioners on how to build real-time applications u...
As organizations scale their AI initiatives on Databricks, a common pattern emerges: one team builds and maintains a vector store, while other teams across different workspaces need to query it. Perha...
Unified stack for geospatial workloads Every day, hundreds of thousands of vessels broadcast their position, speed and heading via AIS. That data stream is a goldmine: commodity traders can sharpen ET...
