Learning to Translate Freehand Design Drawings into CAD Programs – NaturalCAD

Computer Aided Design (CAD) is a multi-billion dollar industry responsible for the digital design of almost all manufactured goods. It leverages parametric modeling, which allows dimensions of a design to be changed, facilitating physically-based optimization and design re-mixing by non-experts. But CAD’s potential is diminished by the difficulty of creating parametric models: in addition to mastering design principles, professionals must learn complex CAD software interfaces. To promote effective modeling strategies and creative flow, design educators advocate freehand drawing as a preliminary step to parametric modeling. Unfortunately, CAD systems do not understand these drawings, so designers must re-create their entire design using complex CAD software. Can we automatically convert freehand drawings to parametric CAD models? Sketch-based modeling techniques do not produce parametric CAD programs; classic CAD reverse-engineering techniques cannot handle drawings as input; the newer field of visual program induction is promising but has been demonstrated only on simple shapes and programs. By leveraging the visual vocabulary shared by drawing and CAD modeling, we will develop a system to translate from the natural language of drawing to the formal language of CAD.

To handle drawings as input, we will treat them as timestamped sequences of strokes, allowing us to cast the problem as one of machine translation from drawing stroke sequences to CAD program token sequences. We observe that drawing strokes are grouped into coherent drawing operations that are correlated with CAD modeling strategies (e.g. first drawing construction lines and simple primitives shapes, then refining). We propose to extract these drawing operations as an intermediate representation, which helps disambiguate between the (potentially infinitely) many programs which can represent a single shape. Performing this extraction and then producing CAD programs are complex search problems; we will leverage novel deep neural networks to guide the search. We propose to gather a paired (drawing, CAD program) dataset from professional designers to help us develop these networks. We will also develop learning algorithms that do not require such ground-truth paired data. Finally, we will develop metrics to assess CAD programs produced by our system, which we will use to evaluate our work and to guide the program search process.