Learning Week II: Industrial skills and advanced topics in ML

The movement of a garment has infinite degrees of freedom. This originates the main difficulties for its robotic handling. Automated manipulation of clothing requires to forecast and understand its position while the clothing is grabbed. We will address these questions and we will show some solutions obtained within the Clothilde project at IRI (CSIC-UPC).

In the era of GPT models, one gets notoriously confronted with the question of how much inductive (geometric) bias can help when scaling up deep neural architectures. In this lecture, we first contrast modern geometric deep learning approaches, such as group equivariant learning, with large-scale Transformer success stories. Secondly, we introduce Clifford (geometric) algebras as an emerging paradigm within the deep learning community. To delve deeper, we commence with a comprehensive overview of modern (plane-based) geometric algebra, which is founded on the representation of isometries as elements within the Pin(p, q, r) group. Subsequently, we propose the concept of group actions and Clifford group equivariant layers. Lastly, we discuss the potential application of these layers when scaling up neural networks, which, in turn, could potentially pave the way for the development of geometric GPTs in the future.

Modern products are increasingly designed, optimised, and manufactured using geometric and physical models held on computer, so-called digital twins. As these continue to grow in scale, complexity and ambition, gaps appear in the digital thread connecting these models, where engineers must manually intervene, and transform their geometry to suit each tool or analysis model. This talk will describe some of the geometry reasoning technology ITI has developed to address these gaps, how hybrid geometry representations have become key to our approach, and where the future opportunities might lie.

In this talk, we explore AI methods for predicting stresses when the parameters describing the shape of the component change. We focus on two techniques, one using mesh deformation and the other using image-based approaches using pretrained convolutional neural networks. We provide numerical results to predict the Van Mises stresses for a parameterized two-dimensional component.

In this work, we introduce ShipHullGAN, a generic parametric modeller built using deep convolutional generative adversarial networks (GANs) for the versatile representation and generation of ship hulls. We trained ShipHullGAN on a large dataset of 52,591 physically validated designs from a wide range of existing ship types, including container ships, tankers, bulk carriers, tugboats, and crew supply vessels. We developed a new shape extraction and representation strategy to convert all training designs into a common geometric representation of the same resolution, as typically GANs can only accept vectors of fixed dimension as input. A space-filling layer is placed right after the generator component to ensure that the trained generator can cover all design classes. During training, designs are provided in the form of a shape-signature tensor (SST) which harnesses the compact geometric representation using geometric moments that further enable the inexpensive incorporation of physics-informed elements in ship design. We have shown through extensive comparative studies and optimisation cases that ShipHullGAN can generate designs with augmented features resulting in versatile design spaces that produce traditional and novel designs with geometrically valid and practically feasible shapes.

Food recognition presents a formidable challenge in computer vision and machine learning, stemming from the intrinsic complexity of food images. The complexity is characterized by high inter-class similarity and intra-class variability, the fine-grained nature of food classes, visual differences arising from different cooking styles, lack of distinctive spatial layouts and rigid structures, and randomly distributed ingredients across the food platter. As is the case with any deep learning problem, the training data plays a pivotal role in food recognition. In this talk, we address two data-centric questions that deepen our comprehension of food data. First, we investigate the issue of data labelling by harnessing self-supervised learning algorithms. We present “All4One”, a novel neighbour-based contrastive SSL approach that employs “centroids” generated through a self-attention mechanism to reduce the distance between different neighbour representations. Second, we tackle the challenge posed by the fine-grained nature of classes through subset learning-based methods. We introduce “Dining on Details”, an expert learning framework for food classification that incorporates lexical information extracted using LLMs. Both proposed solutions shed light on critical aspects of data-centred food recognition and pave the way for more accurate and robust food recognition models.

The ability to generate realistic and diverse 3D shapes has the potential to significantly accommodate the workflow of artists and content creators and potentially enable new levels of creativity through “generative art”. However, despite the impressive progress in generative models that can generate plausible 3D shapes of high fidelity, an important aspect of the generation process is to enable control on what parts of the generated object should change. Therefore, having a basic understanding of the decomposition of an object into parts facilitates controlling what to edit. In the first part of my talk, I will present PartNeRF, which is a part-aware generative model for editable 3D shape synthesis that is trained using posed images. The key idea is to generate objects as a set of locally defined NeRFs, that emerge automatically during training. Our formulation enables part-level control, hence permitting several editing operations such as rigid and non rigid transformations on parts, mixing parts from different objects etc. Although PartNeRF, can reliably edit the appearance and the shape of the generated instance, it is an unconditional generative model. However, being able to generate realistic 3D geometries from versatile user inputs is a crucial task that can unlock numerous editing capabilities. To address this, we introduced PASTA, which is an autoregressive transformer architecture for controllable shape generations that focuses on yielding detailed geometries. PASTA comprises two main components: An autoregressive transformer that generates objects as a sequence of cuboidal primitives and a blending network, implemented with a transformer decoder that composes the sequences of cuboids and synthesizes high quality meshes for each object. Our model is trained in two stages: First we train our autoregressive generative model using only annotated cuboidal parts as supervision and next, we train our blending network using explicit 3D supervision, in the form of watertight meshes. Our model, trained with various conditionings, can be used to perform shape generation from diverse inputs e.g. from scratch, from a partial object, from text and images, as well size-guided generation. Moreover, as PASTA considers the underlying part-based structure of a 3D object, we are able to select a specific part and produce shapes with meaningful variations of this part. Finally, I will present two works for controllable scene synthesis. First, I will talk about ATISS, which is a novel autoregressive transformer architecture for creating diverse and plausible synthetic indoor environments that poses scene synthesis as sequence generation and generates rooms as unordered sets of objects. Next, I will also talk about our recent work, CC3D, which is a conditional generative model for scene synthesis that does not require 3D labelled bounding boxes as supervision and instead can be trained using only posed images.

Within the first year of our life, we develop a common-sense understanding of the physical behavior of the world, which relies heavily on our ability to properly reason about the arrangement of objects in a scene. While this seems to be a fairly easy task for the human brain, computer vision algorithms struggle to form such high-level reasoning. Therefore, the research community shifted their attention to the development of primitive-based methods that seek to represent objects as semantically consistent part arrangements. However, due to the simplicity of existing primitive representations, these methods fail to accurately reconstruct 3D shapes using a small number of primitives/parts. In the first part of my talk, I will introduce several compositional representations that have been explored in the literature. I will address the trade-off between reconstruction quality and number of parts and present Neural Parts, a novel 3D primitive representation that defines primitives using an Invertible Neural Network (INN) which implements homeomorphic mappings between a sphere and the target object. Since a homeomorphism does not impose any constraints on the primitive shape, our model effectively decouples geometric accuracy from parsimony and as a result captures complex geometries with an order of magnitude fewer primitives. Next, I will present our recent work ALTO, that can produce high-fidelity reconstruction of implicit 3D surfaces from noisy point clouds. Our key idea is to alternate back and forth between point and grid latents before converging to a final feature representation that is easy to decode. ALTO is able to yield high-quality 3D meshes while being up to 10 times faster than prior works. Finally, I will present COPILOT, a transformer-based architecture capable of performing collision prediction and localization simultaneously given either multi-view or single-view egocentric video observations. COPILOT accumulates information across multi-view inputs through a novel 4D space-time-viewpoint attention mechanism and provides rich information regarding imminent collisions.

Nowadays, production of goods and industrial components relies on a mixture of digital and classical fabrication steps. AI and ML seem to be right tools to automate the existing workflows to reduce involvement of human beings and optimize the outcome of the manufacturing industry. In this presentation, the audience will be immersed in the existing challenges and opportunities that can be reached with the AI. Generic industrial examples and AI try-outs at ModuleWorks will be presented.

In my talk, I will try to motivate the relevance of the Castelnuovo-Mumford regularity as an invariant that governs the computations with polynomial systems. Namely, I will discuss the relation of this invariant with elimination matrices, Groebner bases, resultants and incremental algorithms for solving polynomial systems. I will also deal with the possible extensions of this invariant to multigraded and sparse systems, and motivate it with many examples coming from applications.

Impressive progress in generative models and implicit representations gave rise to methods that can generate 3D shapes of high quality. However, being able to locally control and edit shapes is another essential property that can unlock several content creation applications. Local control can be achieved with part-aware models, but existing methods require 3D supervision and cannot produce textures.
In the first part of the talk, we present PartNeRF, a novel part-aware generative model for editable 3D shape synthesis that does not require any explicit 3D supervision. Our model generates objects as a set of locally defined NeRFs, augmented with an affine transformation. This enables several editing operations such as applying transformations on parts, mixing parts from different objects etc. Evaluations on various ShapeNet categories demonstrate the ability of our model to generate editable 3D objects of improved fidelity, compared to previous part-based generative approaches that require 3D supervision or models relying on NeRFs.
Existing 3D-aware generative methods – including PartNeRF – are trained using synthetic data, and require multiple views per object instance and known camera poses. However, from a user’s perspective, it would be extremely beneficial to have the ability to generate 3D shapes using only in-the-wild images. In the second part of the talk, we focus on the task of generating 3D shapes from in-the-wild images and present our preliminary results.

Characterizes degenerate cases of cubic plane curves by practicable formulas in terms of the coefficients of the polynomials describing the curve. We will use minors of the well-known matrices associated with the cubic curves such as Hessian matrix.

In the domain of computer vision, decomposing 3D scenes into object instances is a fundamental challenge. In addition, achieving accurate segmentation of 3D scenes without the laborious task of annotating data is paramount for applications spanning robotics to augmented reality. This presentation introduces a paradigm-shifting approach to unsupervised 3D scene segmentation, driven by object-level generative modeling that transcends the necessity for scene-level supervision. By parsing the point cloud of a 3D scene and utilizing the object likelihood of a variational autoencoder model, we discover and segment the objects inside the scene. The empirical validation highlights the effectiveness of our unsupervised 3D scene segmentation approach, paving the way for profound advancements in 3D scene understanding, all without being tethered to the traditional way of exhaustive annotations.

We present new results about the partitioning of measures with polynomials. Namely, for any Borel probability measure $\mu$ on $\R^n$ there exists a set $X\subset \R^n$ of $n+1$ points such that any $n$-variate quadratic polynomial $P$ that is nonnegative on $X$ (i.e. $P(x)\geq 0$, for every $x \in X$) satisfies $\mu\{P\geq 0\}\geq \frac{2}{(n+1)(n+2)}$.
More generally, given an absolutely continuous probability measure $\mu$ on $\R^n$ and $D\leq 2k$, for every $\delta>0$ there exists a set $X\subset \R^n$ with $|X|\leq \binom{n+2k}{n}-n-1$ such that any $n$-variate polynomial $P$ of degree $D$ that is nonnegative on $X$ satisfies $\mu\{P\geq 0\}\geq \frac{1}{\binom{n+2k}{n}+1} – \delta$. These statements are analogues of the celebrated \emph{centerpoint theorem}, which corresponds to the case of linear polynomials. Our results follow from new estimates on the Carath\’eodory numbers of real Veronese varieties, or alternatively, from bounds on the convex symmetric rank of real symmetric tensors.This is a joint work with Alfredo Hubard and Roman Karasev.

Geometrically smooth spline functions are piecewice polynomial functions defined on a mesh, that satisfy properties of differentiability across shared edges; their unstructured nature provides them a wide range of application. We start introducing a G1 spline construction defined by means of smoothing masks on quad meshes approximating the well-known Catmull-Clark surface. Furthermore, we provide a set of G1 basis functions, describe their properties and analyze their space, to be applied in point data fitting problems and Isogeometric Analysis simulations.

The task of identifying an algebraic surface from a single branching curve (apparent contour) can be simplified by recovering a homogeneous equation in four variables from its branching curve (which is defined by the discriminant).
Currently, successful reconstructions have been achieved for smooth surfaces, surfaces with ordinary singularities, ruled surfaces, and the latest work focuses on reconstructing Darboux cyclides.
There is used the fact that the Darboux Cyclides have a singularity along the absolute conic in order to recognize them up to Euclidean similarity transformations.
By applying the same concept, it becomes feasible to reconstruct a planar map from a given branching curve. The stepping stone is the ramification curve, which is obtained as the linear normalization of the branching curve, which lies on a parametric surface. The composition of the parametrization with linear normalization yields a planar map that shares the same branching curve as the parametric surface.

A neural network consisting of piecewise affine building blocks, such as fully-connected layers and ReLU activations, is itself a piecewise affine function supported on a polyhedral complex. This complex has been previously studied to characterize theoretical properties of neural networks, but, in practice, extracting it remains a challenge due to its high combinatorial complexity. A natural idea described in previous works is to subdivide the regions via intersections with hyperplanes induced by each neuron. However, we argue that this view leads to computational redundancy. Instead of regions, we propose to subdivide edges, leading to a novel method for polyhedral complex extraction. A key to this are sign-vectors, which encode the combinatorial structure of the complex. Our approach allows to use standard tensor operations on a GPU, taking seconds for millions of cells on a consumer grade machine. Motivated by the growing interest in neural shape representation, we use the speed and differentiability of our method to optimize geometric properties of the complex.

In this work, we introduce ShipHullGAN, a generic parametric modeller built using deep convolutional generative adversarial networks (GANs) for the versatile representation and generation of ship hulls. At a high level, the new model intends to address the current conservatism in the parametric ship design paradigm, where parametric modellers can only handle a particular ship type. We trained ShipHullGAN on a large dataset of 52,591 physically validated designs from a wide range of existing ship types, including container ships, tankers, bulk carriers, tugboats, and crew supply vessels. We developed a new shape extraction and representation strategy to convert all training designs into a common geometric representation of the same resolution, as typically GANs can only accept vectors of fixed dimension as input. A space-filling layer is placed right after the generator component to ensure that the trained generator can cover all design classes. During training, designs are provided in the form of a shape-signature tensor (SST) which harnesses the compact geometric representation using geometric moments that further enable the inexpensive incorporation of physics-informed elements in ship design. We have shown through extensive comparative studies and optimisation cases that ShipHullGAN can generate designs with augmented features resulting in versatile design spaces that produce traditional and novel designs with geometrically valid and practically feasible shapes.
Video abstract: https://youtu.be/LT9Z52vBgzI

Universal approximation theorems are results that demonstrate the density of a certain class of functions within a space of interest. We will begin our discussion with the early works of Cybenko, Hornik, and others, on feed-forward neural networks, up to more recent results that go beyond the Euclidean setting. Subsequently, we will illustrate how we can use existing universal approximation theorems to prove new ones, even in the light of a certain kind of symmetry.

We discuss the classical methods in curve design such as Bezier and B-Spline. We also discuss some extension to the periodic case and other recent and notable alternatives. The common properties of these tools are that even though they are very intuitive tools, they lack direct control over a curve. On the other hand, the Barycentric rational form can be used to complement the previous methods, since due to the interpolatory property, we can force the curve to pass through specific points. We describe the correspondence of the classical method and the interpolation method to offer new editing possibilities of a curve.

A computational result about recognition of any implicit form of Dupin cyclides will be presented. Applications will include: singularity investigation, illustration of properties of Dupin cyclides suggested in a classical paper of G. Darboux, and investigation of Dupin orthogonal coordinate systems.

Locally Refined B-splines (LR B-splines) [Dokken et al.,Polynomial splines over locally refined box-partitions, CAGD, 2013] have proven to be a very flexible and powerful framework in several applications, such as, geometric modeling, data analysis, visualization and numerical simulation. In this work we extend LR B-splines to allow for non-uniform polynomial degree, where we combine degree elevation with local h-refinement, resulting in local k-refinement, that is, increasing the polynomial degree as well as the smoothness at newly introduced knot lines. The non-uniform degree LR B-splines are scaled by weights to maintain the nestedness of the resulting space and the non-negative partition of unity giving the convex hull property, which can be computed by means of certain two-scale relationship. In this talk, we discuss the theoretical properties of the new spline spaces ensuring the fundamental properties like local support, (scaled) partition of unity, etc., and their application in several 2D model problems, illustrating the efficacy of the proposed adaptive refinement methodology in the context of isogeometric analysis.

Shape reconstruction from raw point sets is a pivotal task in the realms of computer graphics and computer vision. This presentation presents the intricate journey of transforming unstructured point data into coherent 3D representations of objects. We explored both explicit and implicit methods to reconstruct shapes. For explicit reconstruction, we present Bézier primitive segmentation on 3D point clouds. Different from previous primitive segmentations that treat different primitives separately, we take inspiration from Bézier decomposition on NURBS models and transfer it to guide point cloud segmentation casting off primitive types. Experiments show superior performance over previous work in terms of segmentation, with a substantially faster inference speed. For implicit reconstruction, we present a method for reconstructing an isotropic surface triangle mesh from an unoriented point cloud whose density adapts to an estimate of the local feature size (LFS). Instead of performing dense reconstruction followed by remeshing, our approach reconstructs both an implicit function and an LFS-aware mesh sizing function, which are then used for producing the final LFS-aware reconstructed mesh. Our approach’s added value is generating isotropic meshes directly from 3D point clouds with an LFS-aware density. Our experiments demonstrate robustness to noise, outliers, and missing data.