Objaverse: A Universe of Annotated 3D Objects

Abstract

Massive data corpora like WebText, Wikipedia, Conceptual Captions, WebImageText, and LAION have propelled recent dramatic progress in AI. Large neural models trained on such datasets produce impressive results and top many of today’s benchmarks. A notable omission within this family of large-scale datasets is 3D data. Despite considerable interest and potential applications in 3D vision, datasets of high-fidelity 3D models continue to be mid-sized with limited diversity of object categories. Addressing this gap, we present Objaverse 1.0, a large dataset of objects with 800K+ (and growing) 3D models with descriptive captions, tags, and animations. Objaverse improves upon present day 3D repositories in terms of scale, number of categories, and in the visual diversity of instances within a category. We demonstrate the large potential of Objaverse via four diverse applications: training generative 3D models, improving tail category segmentation on the LVIS benchmark, training open-vocabulary object-navigation models for Embodied AI, and creating a new benchmark for robustness analysis of vision models. Objaverse can open new directions for research and enable new applications across the field of AI.

Figure 1: Example instances from our large-scale 3D asset dataset Objaverse. Objaverse 3D assets are semantically diverse, high-quality, and paired with natural-language descriptions.

1 Introduction

Massive datasets have enabled and driven rapid progress in AI. Language corpora on the web led to large language models like GPT-3 [4]; paired image and text datasets like Conceptual Captions [68] led to vision-and-language pretrained models like VilBERT [45]; YouTube video datasets led to video capable models like Merlot-Reserve [87]; and massive multimodal datasets like WebImageText [70] and LAION [67, 66] led to models like CLIP [60] and StableDiffusion [64]. These leaps in dataset scale and diversity were triggered by moving from manually curated datasets to harnessing the power of the web and its creative content.

In contrast to the datasets described above, the size of the datasets we are feeding to our, data-hungry, deep learning models in many other areas of research is simply not comparable. For instance, the number of 3D assets used in training generative 3D models is, maximally, on the order of thousands [24] and the simulators used to train embodied AI models typically have only between a few dozen to a thousand unique scenes [39, 72, 63, 42]. The startling advances brought about by developing large-scale datasets for images, videos, and natural language, demand that an equivalent dataset be built for 3D assets.

We present Objaverse 1.0, a large scale corpus of high-quality, richly annotated, 3D objects; see Fig. 1. Objects in our dataset are free to use and sourced from Sketchfab, a leading online platform for managing, viewing, and distributing 3D models. In total, Objaverse contains over 800K 3D assets designed by over 100K artists which makes this data large and diversely sourced. Assets not only belong to varied categories like animals, humans, and vehicles, but also include interiors and exteriors of large spaces that can be used, e.g., to train embodied agents. Objaverse is a universe of rich 3D data with detailed metadata that can support many different annotations to enable new applications. With this remarkable increase in scale, we see an incredible opportunity for Objaverse to impact research progress across domains. In this work, we provide promising results to answer three questions.

Can 3D vision benefit from a large-scale dataset? First, as a 3D asset resource, Objaverse can support the exciting field of 3D generative modeling. We use data extracted from Objaverse to train generative models for single and multiple categories using GET3D [24] and find that we are able to generate high-quality objects and, moreover, that our generated objects are found by human annotators to be more diverse than those generated by a model trained on ShapeNet objects in 91% of cases.

Can the diversity of 3D models help improve classical 2D vision task performance? To answer this question, we use the diversity of Objaverse to improve the performance of long tail instance segmentation models. Instance segmentation data can be expensive to obtain owing to the cost of annotating contours around objects. The recent LVIS dataset contains annotations for 1,230 categories but the task remains very challenging for present day models, particularly on tail categories that have few examples. We show that increasing the volume of data by leveraging a simple Copy+Paste augmentation method with Objaverse assets can improve the performance of state-of-the-art segmentation methods.

We also use Objaverse to build a benchmark for evaluating the robustness of state-of-the-art visual classification models to perspective shifts. We render objects in Objaverse from random orientations, which is how one might expect to see them in the real world and test the ability of CLIP-style visual backbones to correctly classify these images. Our experiments show that current state-of-the-art models’ performance degrades dramatically in this setting when viewing objects from arbitrary views. Objaverse allows us to build benchmarks to test (and potentially train) for orientation robustness for a long tail distribution of asset categories. Building such benchmarks is made uniquely possible by the scale and diversity of 3D assets in Objaverse. This would simply not be feasible to create in the real world nor can they be generated from existing 2D images.

Dataset	# Objects	# Classes
YCB [5]	77	5
BigBIRD [69]	125	–
KIT [35]	219	145
IKEA [43]	219	11
Pix3D [71]	395	9
GSO [19]	1K	17
EGAD [51]	2K	–
PhotoShape [53]	5K	1
ABO [12]	8K	63
3D-Future [22]	10K	34
ShapeNet [7]	51K	55
Objaverse	818K	21K

Figure 2: Comparison between Objaverse and existing 3D object datasets. (Left:) Visual comparison of instances from Objaverse and ShapeNet for the categories of Car, Bed, Vase, and Bag. Objaverse instances are substantially more diverse since objects can come from many 3D content creation platforms, whereas ShapeNet models look more similar and all come from SketchUp, a 3D modeling platform built for simple architectural modeling. (Right:) Scale comparison table between existing 3D object datasets.

Can a large-scale 3D dataset help us train embodied agents performant embodied agents? We use assets in Objaverse to populate procedurally generated simulated environments in ProcTHOR [17] that are used to train Embodied AI agents. This results in an orders of magnitude increase in the number of unique assets available for use in ProcTHOR scenes (previously limited to AI2-THOR’s [39] asset library of a few thousand unique instances each assigned to one of 108 object categories). Using Objaverse populated scenes enables open vocabulary object navigation from any text description. In this paper, we provide quantitative results for navigating to 1.1K semantic object categories, roughly a 50x increase.

These findings represent just a small fraction of what can be accomplished using Objaverse. We are excited to see how the research community will leverage Objaverse to enable fast and exciting progress in 2D and 3D computer vision applications and beyond.

2 Related Work

Large scale datasets. Scaling the size and scope of training datasets has widely been demonstrated to be an effective avenue of improvement for model performance. In computer vision, the adoption of early large scale datasets such as Imagenet[65, 18] and MS-COCO[44] has dramatically accelerated progress on a variety of tasks including classification, object detection, captioning, and more. Ever since, the diversity and scale of datasets have continued to grow. YFCC100M is a dataset of 99.2M images and 800K videos[77]. OpenImages[40] is a large scale dataset of 9M images that contains labeled subsets bounding boxes, visual relationships, segmentation masks, localized narratives, and categorical annotations. Massive web-scraped datasets containing image-text pairs such as Conceptual Captions[68], WIT[70], and LAION[67, 66] have seen increased popularity recently as they have been used to train impressive models for vision-language representation learning[60, 29, 32], text-to-image generation[62, 61, 64, 32], and vision-language multitasking[10, 73, 79, 9].

3D datasets. Current large-scale 2D image datasets offer three crucial components that benefit learning: scale, diversity, and realism. Ideally, models that reason about 3D objects should have access to datasets that meet these same criteria. However, of the numerous 3D object datasets that currently exist, none are able to excel in all three categories to the same degree as their 2D counterparts. Datasets such as KIT[35], YCB[5], BigBIRD[69], IKEA[43], and Pix3D[71] provide image-calibrated models over a diverse set of household objects, but severely lack in scale with only a few hundred objects at most. EGAD[51] procedurally generates 2K objects for grasping, but produces objects that are not that realistic or diverse. Slightly larger datasets of photo-realistic objects include GSO[19], PhotoShape[53], ABO[12] and 3D-Future[22], and ShapeNet[7] with object counts in the tens of thousands, see Fig. 2 for comparisons between Objaverse and these datasets. Datasets for CAD models, such as ModelNet[83] and DeepCAD[82], and ABC[38] do not include textures or materials, which limits their ability to represent objects that could plausibly be found in the real world. Datasets of scanned 3D objects and environments are valuable for real-world understanding[14, 15, 11, 41], but are quite small and limited. In addition to containing numerous artist designed objects, Objaverse contains many scanned assets, making it a useful source of data for learning from real-world distributions.

While rapid progress has been made in developing datasets that combine image and text, in contrast, only a few datasets that pair language and 3D data exist. Text2Shape[8] released a dataset of 15,038 chairs and tables from ShapeNet each with around 5 text captions, giving 75,344 total text-shape pairs. ShapeGlot[1] released the CiC (Chairs in Context) dataset which contains 4,511 chairs from ShapeNet along with 78,789 descriptive utterances generated from a referential game. Due to the small scale and limited diversity of these datasets, current SoTA text-to-3D models[47, 57, 31] forgo the use of 3D datasets entirely and instead rely on 2D image-text supervision.

3 Objaverse

Objaverse is a massive annotated 3D dataset that can be used to enable research in a wide range of areas across computer vision. The objects are sourced from Sketchfab, an online 3D marketplace where users can upload and share models for both free and commercial use. Objects selected for Objaverse have a distributable Creative Commons license and were obtained using Sketchfab’s public API. Aside from licensing consideration, models marked as restricted due to objectionable or adult thematic content were excluded from the dataset.

(a) Breakdown of high-level Objaverse categories

(b) Word cloud of Objaverse metadata tags.

(c) Objaverse-LVIS category occurrence distribution.

(d) Objaverse tag occurrence distribution.

Figure 3: Objaverse statistics. (a) All 18 high-level categories present in Objaverse’s metadata with their corresponding number of occurrences. The relative share of most popular categories are evenly split, with a small number of less frequently categories. (b) A sample of several thousand popular object tags found in Objaverse log-scaled by their frequency. (c) A histogram of fine-grained Objaverse-LVIS categories with representative members from several bins highlighted. (d) A histogram of Objaverse tags with representative members from several bins highlighted (note y-axis log scale). Tags from the low-occurrence side of the distribution correspond to unique objects that, taken individually, are rarely seen in the world. Frequently used tags like ”furniture” and ”car” reflect their real-world normalcy, but the high frequency of assets like ”sword” diverge from their real-world counterparts.

Model metadata. Objaverse objects inherit a set of foundational annotations supplied by their creator when uploaded to Sketchfab. Figure 4 shows an example of the metadata available for each model. The metadata includes a name, assignments to a set of fixed categories, a set of unrestricted tags, and a natural language description.

Objaverse-LVIS. While Objaverse metadata contains a great deal of information about objects, Sketchfab’s existing categorization scheme covers only 18 categories, too coarse for most applications. Object names, categories, and tags provide multiple potential categorizations at varying levels of specificity and with some inherent noise. However, for many existing computer vision tasks, it is useful to assign objects to a single category drawn from a predetermined set of the right size and level of semantic granularity.

We choose the categories from the LVIS dataset [26] for categorizing a long-tail subset of objects in Objaverse. We construct a 47K LVIS categorized object subset, called Objaverse-LVIS, comprised of objects uniquely assigned to one of 1156 LVIS categories. We perform these assignments by first selecting 500 candidate objects per category using a combination of predictions from a CLIP classification model and candidates suggested by terms in their metadata. This combined pool contains objects visually resembling the target category (from the CLIP features of their thumbnail images) that might have missing metadata, as well as visually unusual instances of a category that are accurately named or tagged. These 250k candidate objects were then manually filtered and their assigned categories verified by crowdworkers. Since we only presented 500 object candidates per class, many popular categories, such as chair or car, have substantially more objects that could be included in Objaverse-LVIS with future annotations.

Animated objects and rigged characters. Objaverse includes 44K animated objects and over 63K objects self-categorized as characters. Examples of animations include fridge doors opening, animals running, and the hands on a clock moving. Rigged characters can be set up for animation and rendering, and may often come annotated with bone mappings. The vast scale of animations available in Objaverse can support a wide range of research in temporal 3D learning, such as building text-based animation generative models [76], representing object changes over time with NERFs [59, 54], and temporal self-supervised learning via. future frame prediction [87, 30].

Figure 4: An example of metadata available for each object in Objaverse. Each uploaded object has a 3D model, user-selected rendered thumbnail image, name, description, tags, category, and stats, among additional metadata.

Figure 5: Highlights of the visual diversity of objects that appear in Objaverse, including animated objects, rigged (body-part annotated) characters, models separatable into parts, exterior environments, interior environments, and a wide range visual styles.

Articulated objects. Decomposing 3D objects into parts has led to a flurry of research in the past few years, including work in learning robotic grasping policies [86, 84], 3D semantic segmentation [50], and shape generation [49]. Since many objects in Objaverse were uploaded by artists, the objects often come separated into parts. Figure 5 shows an example, where a chair is separated by its backrest, wheels, and legs, among many smaller parts.

Exteriors. Photogrammetry and NERF advances have enabled the commercialization of capturing high-quality 3D objects of large exteriors by taking pictures  [75, 85]. In Objaverse, there are a large number of scanned buildings, cities, and stadiums. Figure 5 shows an example of a 3D object of NYC’s skyline captured through a scan.

Objaverse-Interiors. There are 16K+ interior scenes in Objaverse, including houses, classrooms, and offices. The scenes often have multiple floors, many types of rooms, and are densely populated with objects from human input. Objects in the scenes are separable into parts, which allows them to be usable for interactive robotics, embodied AI, and scene synthesis. To put the scale of Objaverse-Interiors in perspective, the number of scenes in Objaverse-Interiors is significantly larger than the 400 or so existing hand-built interactive embodied AI scenes [39, 23, 42, 72].

Visual styles. Objects in the world can be constructed in many styles and often differ in style based on the time-period, geographic location, and artist’s style. Objaverse objects cover a vast set of visual styles, including 3D scans, 3D modeled objects from virtually any platform, point clouds, and photo-realism via physically based rendering (PBR) [56]. Moreover, instances of objects often appear with many styles, which is critical for training and evaluating robust computer vision models [60]. Figure 5 shows examples of chairs in Objaverse in many different styles, including Gothic, modern, Victorian, cartoon, and abstract.

Statistics. Objaverse 1.0 includes 818K 3D objects, designed by 160K artists. There are ${>}$2.35M tags on the objects, with ${>}$170K of them being unique. We estimate that the objects have coverage for nearly 21K WordNet entities [48] (see appendix for details). Objects were uploaded between 2012 and 2022, with over 200K objects uploaded uploaded just in 2021. Figure 3 visualizes several statistics of the dataset, including the breakdown of objects into their self-assigned Sketchfab categories, a word cloud over the tags, a frequency plot of the tags, and the number of objects in Objaverse-LVIS categories.

4 Applications

In this section, we present 4 initial distinct applications of Objaverse, including 3D generative modeling, instance segmentation with CP3D, open-vocabulary ObjectNav, and analyze robustness in computer vision models.

4.1 3D Generative Modeling

(a) Comparison of Bags generated with Objaverse and ShapeNet.

(b) Shoe and Fruit&Veg. generations.

(c) Fruit&Veg. interpolation.

Figure 6: (a) Example GET3D Bag object generations using Objaverse and ShapeNet models for training. (b) Additional Shoe and Fruit&Veg generations from Objaverse models. (c) models generated when interpolating between two, randomly sampled, latent encodings with our trained Fruit&Veg. model; what appears to be a pumpkin smoothly transforms into a mushroom.

3D generative modeling has shown much improvement recently with models such as GET3D [24] delivering impressive high quality results with rich geometric details. GET3D is trained to generate 3D textured meshes for a category and produces impressive 3D objects for categories like Car, Chair, and Motorcycle using data from ShapeNet [7]. Objaverse contains 3D models for many diverse categories including tail categories which are not represented in other datasets. It also contains diverse and realistic object instances per category. This scale and diversity can be used to train large vocabulary and high quality 3D generative models. In this work, we showcase the potential of this data as follows. We choose three categories of objects, Shoe, Bag, and Fruit&Veg, and subsample objects from Objaverse to create three separate datasets containing, respectively, 143 shoes, 816 bags, and 571 fruits & vegetables (116 apples, 112 gourds, 92 mushrooms, 68 bananas, 52 oranges, 52 pears, 31 potatoes 24 lemons, and 24 pineapples). For comparison, we also train a GET3D model on the set of 83 bags from the ShapeNet dataset. Fig. 6 shows a collection of 3D objects generated by our trained GET3D models. Qualitatively, the 3D-meshes generated by the Objaverse-trained models are high-quality and diverse, especially when compared to the generations from the ShapeNet-trained model. To quantify this observation, we asked crowdworkers to rate the diversity of Bag generations produced by the Objaverse and ShapeNet trained models. When shown collections of nine randomly sampled generations from both models, workers rated the collection generated from the Objaverse trained model as more diverse in appearance 91% of the time.

Our fruits and vegetables, composed of 9 varieties produces perhaps the highest quality output, a promising signal that can inspire future work in text-to-3D generation.

4.2 Instance Segmentation with CP3D

Figure 7: An illustration of 3DCP (3D copy-paste) for segmentation augmentation. We render 3D objects from multiple views and paste them over LVIS training images.

Method	AP	APr	APc	APf
RFS [26]	23.7	13.3	23.0	29.0
EQLv2 [74]	25.5	17.7	24.3	30.2
LOCE [21]	26.6	18.5	26.2	30.7
NorCal with RFS [52]	25.2	19.3	24.2	29.0
Seesaw [78]	26.4	19.5	26.1	29.7
GOL [3]	27.7	21.4	27.7	30.4
GOL + 3DCP	28.3	21.8	28.3	31.1

Table 1: Comparison of our approach (GOL+3DCP) against SoTA Mask-RCNN ResNet-50 models on LVIS. We report results for APr, APc, and APf which measure AP for categories that are rare (appear in 1-10 images), common (appear in 11-100 images), and frequent (appear in $>$100 images), respectively

Figure 8: An existing ProcTHOR scene (left) and a semantically similar ProcTHOR generatable scene with Objaverse objects (right).

A key advantage of using simulated data for computer vision is that it is much cheaper to obtain expert annotations. Annotated Objaverse objects can be rendered into images, allowing them to serve as a rich source of additional data that can be used to enhance model performance on 2D computer vision tasks. As a proof-of-concept demonstrating the effectiveness of this approach, we use segmented data from Objaverse objects as auxiliary labels for training models on the LVIS dataset for Large Scale Instance Segmentation [26]. The LVIS dataset contains instance segmentation masks for 1200 object categories that occur throughout a set of 164k images. Recognition is especially challenging in this task due to the long tail of the object category distribution in this dataset. LVIS categories only contain an average 9 instances across the dataset, so training on simulated data is a promising approach for overcoming the challenges of learning in this low-sample regime.

Using the LVIS-annotated subset of Objaverse, we introduce 3DCP: an enhancement to the simple, but effective, copy-and-paste technique of[25]. Figure 7 shows an example of the setup for 3DCP. Here, we render different views of 3D objects and paste them on-top of existing LVIS images. We render 5 distinct views of each object and cache them for use throughout training. During training, an image is selected for the copy-paste augmentation with 0.5 probability, and once selected, 1-3 images of randomly chosen LVIS-annotated Objaverse objects are pasted onto the selected training image. The segmentation masks of the selected objects are added to the training image’s annotation as well. Object images and masks are randomly scaled and translated before being pasted. We use this strategy to finetune the pretrained ResNet-50 Mask-RCNN [27, 28] of [3]. As shown in Tab.1, simply finetuning this model for 24 epochs yields performance gains across several metrics.

4.3 Open-Vocabulary ObjectNav

In this section, we introduce open-vocabulary ObjectNav, a new task propelled by the vast diversity of objects that appear in Objaverse. Here, an agent is placed at a random starting location inside of a home and tasked to navigate to a target object provided from a text description (e.g. “Raspberry Pi Pico”). To facilitate this task, we procedurally generate 10K new homes in ProcTHOR [17] fully populated with objects from Objaverse-LVIS. Until now, ObjectNav tasks have focused on training agents to navigate to 20 or so target objects provided their category label [16, 63, 17], and existing interactive embodied AI simulations, including ProcTHOR, only include around 2K total objects across around 100 object types [42, 17, 72]. In this work, we take a large step to massively scale the number of target objects used in ObjectNav (20 $\to$ OpenVocab), the number of objects available in simulation (2K $\to$ 36K), and the number of object types of the objects (100 $\to$ 1.1K).

Object placement. To make the placement of objects in the houses more natural, we use the Objaverse-LVIS subset and annotate placement constraints for each object category. Specifically, we annotate if objects of a given category typically appears on the floor, on-top of a surface, or on a wall. If instances of the object category may appear on the floor, we also annotate whether it may appear in the middle of the scene (e.g. a clutter object like a basketball) or on the edge of the scene (e.g. a toilet or a fridge). For objects placed on the floor, we also to automatically detect flat regions on top of the object’s mesh to place surface object types. The annotations are used by ProcTHOR for sampling objects to place in a scene. We also filter out Objaverse-LVIS objects that do not appear inside of homes, such as a jet plane. Structural objects, like doors and windows, are inherited from ProcTHOR as they would require additional cleanup.

Object size correction. Objects in Sketchfab may be uploaded at unnatural scales (e.g. a plant being as large as a tower). We therefore scale the objects to be of a reasonable size for them to look natural in a house. Here, for each object category, we annotate the maximum bounding box dimension length that every instance of the object category should be scaled to. For example, we annotate the maximum bounding box dimension for bookcase to be 2 meters and fork to be 0.18 meters. If a 3D modeled bookcase then has a bounding box of 20m$\times$6m$\times$3m, we shrink each side by a factor of $\max(20,6,3)/2=5$.

Preprocessing for AI2-THOR. We add support to AI2-THOR for loading objects on the fly at runtime. Previously, all objects had to be stored in a Unity build, but such an approach is impractical when working with orders of magnitude more object data. For each object, we compress it with Blender [13] by joining all of its meshes together, decimate the joined mesh such that it has at most 5K vertices, and bake all the UV texture maps into a single texture map. We then generate colliders using V-HACD [46] to support rigid-body interactions.

Approach. Given procedural houses populated with Objaverse-LVIS, the task is to navigate to the proximity of a chosen target object and invoke a task-completion action when the target object is in sight, given an open-vocabulary description formed with the template “a {name} {category}”. The name is the object name given by its creator, which is often descriptive. We filter each by whether it is detected as being written in English by a language detector [34, 33], and fall back to a class-only description for non-English name. Examples of the possible expressions include “a victorian-monobike motorcycle”, “a unicorn pony”, or “a dino ghost lizard”. The agent, similar to the ones in [36], observes an RGB egocentric view of the environment, pre-processed by the visual branch of a frozen ResNet-50 CLIP model [60] – the target description is pre-processed by the corresponding text branch. We train the agent with DD-PPO [81] and evaluate on houses with floor plans, objects, and descriptions unseen in training. We use the AllenAct [80] framework to train our agent. Our trained agent achieves a success rate of 19.9%, for a random policy success of 5.1%. For more details about the experiment refer to the appendix.

4.4 Analyzing Robustness

Figure 9: Examples of objects rendered from random orientations and their 0-shot classification categories with the CLIP ViT-B/32.

A persistent bias present in many image datasets, e.g. ImageNet [65], is that the subjects of interest are generally photographed from a forward-facing, canonical, orientation. When, for example, taking a photograph of a television, few would choose to take this photograph crouched on the floor behind the television. This impact of this bias was studied by Alcorn et al. [2] who find that modern computer vision systems are highly susceptible to deviations from canonical poses. This is more than a theoretical problem: computer vision systems deployed in the real world will frequently encounter objects in non-canonical orientations and in many applications, e.g. autonomous driving, it will be safety critical that they behave well.

Given the above, we adopt the experimental design of Alcorn et al. and design, using Objaverse assets, a benchmark for evaluating the robustness of state-of-the-art computer vision classification models to orientation shifts. In particular, for each object in our Objaverse-LVIS subset, we render 12 images of the object from random orientations rendered upon a background with RGB values equalling the mean RGB values from ImageNet; see Fig. 9 for examples. This ability to, at scale, render objects from random viewpoints is a practical impossibility in the real world but is made trivial when using 3D assets. We then evaluate several modern open-domain image-classification networks (constrained to the ${\approx}1{,}200$ LVIS categories) on these images and report 4 metrics for each model. These metrics include:

	Random Rotation		Any Rotation
Model	Top-1	Top-5	Top-1	Top-5	$\Delta$Top-1
OpenAI-400M [60]
RN50	21.4%	45.0%	43.9%	70.8%	22.5%
ViT-L/14	29.1%	54.5%	52.3%	77.2%	23.2%
LAION-400M [67]
ViT-B/32	24.1%	48.5%	46.9%	74.2%	22.8%
ViT-L/14	30.6%	56.8%	50.5%	77.0%	19.9%
LAION-2B [66]
ViT-B/32	27.0%	51.8%	50.3%	76.1%	23.3%
ViT-L/14	32.9%	59.2%	52.1%	78.0%	19.2%
ViT-H/14	32.3%	58.8%	50.1%	77.3%	17.8%

Table 2: Evaluating 0-shot CLIP classification models on our rotational robustness benchmark. $\Delta$Top-1 denotes the difference between Top-1 Any Rotation and Top-1 Random Rotation. Models are strongly overfit to standard views of objects.

$\bullet$ Top-1 Random Rotation – the frequency with which a model correctly classifies an image as belonging to the respective LVIS category.

$\bullet$ Top-1 Any Rotation – the frequency with which a model classifies an image correctly from at least one of the 12 random orientations.

This second metric is diagnostic and serves to represent a model’s performance when shown an object from a canonical pose. We also have Top-5 variants of the above metric where the correct category need only be in the top 5 predictions from the model. We report our results in Tab. 2 in which we evaluate a variety of performant pretrained models. Comparing the gap in performance between the Top-$k$ Random Rotation and Top-$k$ Any Rotation metrics we find that model performance dramatically degrades when viewing objects from unusual orientations.

5 Conclusion

We present Objaverse, a next-generation 3D asset library containing 818K high-quality, diverse, 3D models with paired text descriptions, titles, and tags. As a small glimpse of the potential uses of Objaverse, we present four experimental studies showing how Objaverse can be used to power (1) generative 3D models with clear future applications to text-to-3D generation, (2) improvements to classical computer vision tasks such as instance segmentation, (3) the creation of novel embodied AI tasks like Open Vocabulary Object Navigation, and (4) quantifying the rotational robustness of vision models on renderings of objects. We hope to see Objaverse enable a new universe of new applications for computer vision.

Appendix A Instance Segmentation with CP3D

Model.

We use the Mask-RCNN[27] model of [3] with a ResNet-50 backbone[28]; no additional changes to their model are made. Instead of a softmax activation, the model uses a Gumbel activation, given by the formula $\eta(q)=\exp(-\exp(-q))$, to transform logits into probabilities. More details about the model and activation can be found in [3].

Training.

We take the pretrained ResNet-50 Mask-RCNN checkpoint of [3] and finetune the model for 24 epochs with the CP3D augmentation integrated into the training pipeline. We use a batch size of 64 and a learning rate of $0.002$.

Additional Results

Here we report detection metrics in addition to the segmentation results reported in the paper in Table 1. Notably, we see an impressive gain of two points on AP for rare categories.

Method	AP	APr	APc	APf
GOL [3]	27.5	19.8	27.2	31.2
GOL + 3DCP	28.9	21.8	28.7	32.2

Table 3: Detection results for bounding box AP category metrics. APr, APc, and APf measure AP for categories that are rare (appear in 1-10 images), common (appear in 11-100 images), and frequent (appear in $>$100 images), respectively.

Appendix B Open-Vocabulary ObjectNav

Model.

The agent’s embodiment is a simulated LoCoBot [6]. The action space consists of six actions: MoveAhead, RotateLeft, RotateRight, End, LookUp, and LookDown. Given the excellent exploration capabilities of EmbCLIP [36, 17], we opt to keep the same overall architecture, just replacing the learned embedding for target types in prior work by a linear projection of the text branch output of CLIP for the target description, as shown in Fig. 10. Additionally, in order to provide more information about the target and the current visual input, we increase the respective internal representations for each modality from the original 32-D to 256-D. Note that our model does not employ the alternative zero-shot design described in [36], where the target description is not observed by the agent’s RNN. Given the scale of Objaverse-LVIS, we can train agents with good generalization following a more standard design.

Figure 10: Open-Vocabulary ObjNav Model overview. The ObjectNav model (employing an RNN) uses the high-level architecture illustrated here, where it receives features from the visual and target object description encoders, besides previous hidden units and actions as input, and outputs the next action.

Training.

For training, we use ProcTHOR to procedurally generate 10,080 houses. Each house has up to three rooms, entirely populated with Objaverse-LVIS assets except for structural components like doors and windows, which are inherited from ProcTHOR [17]. We sample targets corresponding to LVIS categories for which a single instance is present in the scene, resulting in a total of 9,421 unique assets corresponding to 262 categories targeted during training. Training uses DD-PPO [81] and is distributed across 28 GPUs on 7 AWS g4dn.12xlarge machines, with each GPU hosting 360 houses and the subset of Objaverse-LVIS assets populating them. The training hyperparameters, identical to the ones in [17], and the 262 training target categories are listed in Table 4 and Table 5, respectively.

Hyperparameter	Value
Discount factor ($\gamma$)	0.99
GAE parameter ($\lambda$)	0.95
Value loss coefficient	0.5
Entropy loss coefficient	0.01
Clip parameter ($\epsilon$)	0.1
Rollout horizons	32, 64, 128
Rollout timesteps	20
Rollouts per minibatch	1
Learning rate	$3\cdot 10^{-4}$
Optimizer	Adam [37]
Gradient clip norm	0.5

Table 4: Training hyperparameters for Open-Vocabulary ObjectNav.

Bible, Christmas tree, Rollerblade, alligator, ambulance, amplifier, arctic (type of shoe), armor, banner, barbell, barrel, barrow, baseball bat, basketball, bat (animal), bath mat, beachball, bear, bed, beetle, bench, beret, bicycle, binder, binoculars, bird, blackberry, bookcase, boot, bottle, bowling ball, bullhorn, bunk bed, bus (vehicle), butterfly, cab (taxi), cabinet, canoe, cape, car (automobile), card, cardigan, carnation, cart, cassette, cat, chair, chaise longue, chicken (animal), clothes hamper, coatrack, coffee table, cone, convertible (automobile), cornice, cow, cowboy hat, crab (animal), crate, crossbar, cube, cylinder, deck chair, deer, desk, dinghy, dirt bike, dog, dollhouse, doormat, dove, drawer, dresser, duckling, dumbbell, dumpster, easel, elephant, elk, fan, ferret, file cabinet, fireplace, fireplug, fishing rod, flag, flagpole, flamingo, flip-flop (sandal), flipper (footwear), foal, football (American), footstool, forklift, frog, futon, garbage, gargoyle, giant panda, giraffe, golf club, golfcart, gondola (boat), goose, gorilla, gravestone, grill, grizzly, grocery bag, guitar, handcart, hat, heater, hockey stick, hog, horse, horse carriage, jeep, kayak, keg, kennel, kitchen table, kitten, knee pad, ladder, ladybug, lamb (animal), lamp, lamppost, lawn mower, legging (clothing), lion, lizard, locker, log, loveseat, machine gun, mailbox (at home), manhole, mascot, mast, milk can, minivan, monkey, mop, motor, motor scooter, motor vehicle, motorcycle, mushroom, music stool, nut, ostrich, owl, pajamas, parasail (sports), parka, penguin, person, pet, pew (church bench), piano, pickup truck, pinecone, ping-pong ball, playpen, pole, polo shirt, pony, pool table, power shovel, propeller, pug-dog, pumpkin, rabbit, radiator, raincoat, ram (animal), rat, recliner, refrigerator, rhinoceros, rifle, road map, rocking chair, router (computer equipment), runner (carpet), saddle (on an animal), saddle blanket, saddlebag, sandal (type of shoe), scarecrow, scarf, sculpture, seabird, shark, shepherd dog, shield, shirt, shoe, sink, skateboard, ski parka, skullcap, snake, snowmobile, soccer ball, sock, sofa, sofa bed, solar array, sparkler (fireworks), speaker (stero equipment), spear, spider, sportswear, statue (sculpture), step stool, stepladder, stool, subwoofer, sugarcane (plant), suit (clothing), suitcase, sunhat, surfboard, sweat pants, sweater, swimsuit, table, tape measure, tarp, telephone pole, television camera, tennis ball, tennis racket, tights (clothing), toolbox, tote bag, towel, trailer truck, trampoline, trash can, tricycle, trousers, truck, trunk, turtle, tux, underdrawers, vacuum cleaner, vending machine, vest, wagon wheel, water ski, watering can, wet suit, wheel, window box (for plants), wok, wolf, and wooden leg.

Table 5: Training target types for Open-Vocabulary ObjectNav.

Testing.

For testing, we sample 150 episodes for each of 30 target categories, which are a subset of the training target categories. The resulting 4,500 episodes are sampled from 151 procedural houses not seen during training. The 30 testing target categories are listed in Table 6. For the results provided in the main paper, the agent is trained for just 18 million simulation steps, but the resulting policy already shows reasonable performance given the variety of targets and scenes. Improved performance can be achieved with extended training (e.g., after approx. 460 million steps, the success rate is 33.0%).

Christmas tree, bed, bench, blackberry, chair, chicken (animal), dog, easel, elk, fireplug, forklift, garbage, gargoyle, guitar, mascot, motor, penguin, pony, pool table, radiator, rifle, scarf, sock, speaker (stero equipment), sportswear, sweat pants, trash can, trunk, wet suit, and wheel.

Table 6: Testing target types for Open-Vocabulary ObjectNav.

Appendix C Composition

Human subjects data.

A portion of the data included in Objaverse is generated by human subjects (i.e. crowdworkers recruited through Amazon’s Mechanical Turk platform) as outlined in Section 3 and detailed below. The collection process has been reviewed and approved for release by an Institutional Review Board.

Data collection interfaces.

Human annotators were used to provide the category labels for Objaverse-LVIS as described in Section 3. This task was accomplished by first creating sets of 500 candidate objects for each LVIS category. These candidate sets included objects visually resembling the target category (as ranked by the CLIP features of their thumbnail images), as well as instances whose metadata contained terms with a high similarity to the target category (as ranked by their GloVe vector similarity [55]). Candidate objects were shown to crowdworkers nine at a time, and they were asked to mark objects that were members of the category, as shown in Figure 11 a. In addition to the visual reference for each object, annotators also had access to the object’s name and were encouraged to use this when helpful. Human annotators were also used to rate the relative diversity of of 3D objects generated by models trained using Objaverse and ShapeNet. The user interface and instructions for this task are shown in Figure 11 b. Two sets of nine objects generated by each model were shown with random left-right orientations, and workers were asked to choose the set exhibiting the greater variety in appearance.

(a) Screenshot of Objaverse-LVIS categorization task.

(b) Screenshot of relative diversity rating task.

Figure 11: Data collection interfaces.

Appendix D Estimating Coverage

We use OpenAI’s CLIP ViT-B/32 model to estimate the categorical coverage of the objects in Objaverse. Specifically, for each object, we compute the CLIP image embedding from the thumbnail and the cosine similarity between an text embedding of each WordNet entity [20]. The entity with the maximum cosine similarity is then assigned as the object’s entity. The WordNet entities are textually encoded in the form, “a {entity} is a {definition}”, which is loosely inspired by CuPL [58]. For instance, we might have “a bat is a nocturnal mouselike mammal with forelimbs modified to form membranous wings and anatomical adaptations for echolocation by which they navigate” or “a bat is a club used for hitting a ball in various games”. Computing the nearest WordNet entity for each object gave us an estimated coverage of 20.8K entities.