Having a deep understanding of the low-level details of data science techniques can be highly beneficial for data professionals in several ways.
Firstly, it allows them to have a deeper understanding of the underlying principles and assumptions of the techniques they are using. This understanding enables them to choose the right tool for the job and to adapt the technique to the specific context they are working on. For instance I majored in spatial econometrics, a set of techniques to deal with spatial autocorrelation. I have not been able to use them at scale but they provide me the intuition to do spatial feature engineering in some of the projects I worked.
Secondly, a deeper understanding of low-level details allows data scientists to break down complex problems into smaller, more manageable pieces. This helps them to identify the most relevant parts of a technique for their specific use case and to adapt it as necessary. For example, having an understanding of the inner workings of deep learning models can help a data scientist to identify the most relevant layers and parameters for a specific project.
Thirdly, having a deep understanding of data science techniques helps data professionals to avoid becoming “library practitioners” who rely solely on pre-existing libraries and packages without much thought. While libraries can be useful, they are not always a perfect fit for every use case. A deep understanding of the underlying principles and assumptions of a technique can help a data scientist to make informed decisions about when and how to use libraries, and to adapt them to their specific needs.
However, it is important to note that having a deep understanding of data science techniques is not a guarantee of success. It is important to ensure that these skills are aligned with the specific needs and goals of a project or organization. For instance, being an expert in deep learning may not be useful in a company that does not use this technology. Ultimately, the key is to strike a balance between depth of knowledge and relevance to the specific context in which a data professional operates.