Ludwig 0.8: A Novel and Efficient LLM

October 3, 2023


Ludwig is an open-source toolkit for building and fine-tuning custom machine learning models without writing code. It is a popular choice for building chatbots, virtual assistants, and other text-based applications. Developed and open-sourced by Uber in 2019, Ludwig is a low-code framework designed to simplify the process of building and deploying custom AI models. 

With its declarative YAML configuration files, Ludwig allows users to train state-of-the-art models without the need for intricate coding or deep understanding of machine learning algorithms. Whether you're looking to build large language models (LLMs), text classifiers, or even multi-modal models that combine text, images, and other data types, Ludwig offers a streamlined, user-friendly approach.

It is easy to define deep learning pipelines with a simple and flexible data-driven configuration system. It supports a wide range of NLP tasks, including text classification, question answering, and summarization. Ludwig also supports the training and fine-tuning of LLMs. This makes it a powerful tool for building custom LLMs that are tailored to your specific needs. The rise of large language models like OpenAI's GPT-3 and Meta's LLaMA-2 has created a demand for tools that can fine-tune these models for specific tasks or industries. Ludwig's latest release, version 0.8, addresses this need head-on by introducing features that make it easier to customize LLMs for specific applications, thereby making the technology more accessible and applicable to real-world problems.

Why Build Your Own LLM?

A user might find several compelling reasons to build their own Language Learning Model (LLM):

  • To enhance performance and accuracy tailored to specific tasks. Pre-trained LLMs often rely on general-purpose datasets, which may not be perfectly aligned with unique requirements. Creating an LLM from scratch using one’s own data can yield better performance and accuracy for specific tasks.
  • To safeguard data privacy. Utilizing a pre-trained LLM, which is trained on large, diverse datasets, could potentially expose sensitive or confidential information. By building one’s own LLM, the user can ensure that their data remains private and secure.
  • To customize the LLM according to specific needs. Pre-trained LLMs are generally designed for broad applications and may lack certain features or capabilities that are required. Building one’s own LLM allows for customization to meet these specific needs.

Benefits of Using Ludwig 0.8 to Build LLMs

Ludwig 0.8 is a beneficial tool for constructing LLMs for several reasons:

  • User-Friendly: Ludwig employs a declarative YAML configuration file, making it straightforward to build and train models without the need for extensive coding.
  • Versatility: Ludwig can be used for a variety of Natural Language Processing (NLP) tasks, such as text classification, question answering, and machine translation. It also supports multi-modal learning, allowing the user to train models that utilize both text and other modalities like images and audio.
  • Comprehensive Configuration Validation: Before the training process begins, Ludwig validates the configuration file to detect any invalid parameter combinations, thereby preventing runtime failures.
  • Scalability and Efficiency: Ludwig comes with features optimized for large-scale models and datasets, including automatic batch size selection, distributed training, and parameter-efficient fine-tuning.

Brief History

Ludwig was born out of a need to simplify  machine learning and make it accessible to people without a deep technical background in the field. Uber, the company behind Ludwig, initially developed the framework to solve its internal data challenges. Recognizing its potential for broader applications, Uber decided to open-source Ludwig in 2019, making it available for developers, data scientists, and businesses worldwide.

The initial release was groundbreaking in many ways. It offered a low-code, highly flexible framework that allowed users to build machine learning models using simple YAML configuration files. The framework was designed to be agnostic to the type of data and the task, providing a level of flexibility that was not commonly seen in other machine learning frameworks at the time. The initial release focused on tasks like text classification, image recognition, and even time-series forecasting, among others.

Ludwig 0.7: A Recap

In version 0.7, Ludwig made significant advancements by introducing support for large pretrained models, including large language models (LLMs). This version was optimized for efficiency, featuring automatic batch size adjustments and more efficient data loading mechanisms. It also focused on enhancing its capabilities for Predictive AI tasks like classification and regression. To make the framework more accessible, the documentation and tutorials were revamped, providing a smoother experience for both beginners and experts.

However, Ludwig 0.7 had its limitations. It was primarily geared towards Predictive AI tasks, with limited support for Generative AI tasks like text generation and chatbots. While it introduced some optimizations for large models, scalability was still a concern, especially for models too large for a single GPU or node. Additionally, it lacked advanced fine-tuning capabilities and had limited features for multi-modal learning.

Ludwig 0.8

Ludwig 0.8 brings a host of new features aimed at enhancing the user experience and expanding its capabilities. Notable improvements include enhanced support for Large Language Models with a new ‘LLM’ model type, declarative fine-tuning options, and integration with Deepspeed for efficient parallel training. These updates address some of the limitations of Ludwig 0.7, making the framework even more versatile and user-friendly.

Core Features and Capabilities

Declarative Model Configuration

One of Ludwig's standout features is its declarative model configuration. Users can define their models using a simple YAML file, specifying input and output features, types of encoders and decoders, and various hyperparameters. This eliminates the need for writing extensive code, making the model-building process more straightforward and accessible.

Multi-Modal and Multi-Task Learning

Ludwig supports multi-modal learning, allowing users to build models that can process multiple types of data (text, images, numerical data, etc.) simultaneously. It also supports multi-task learning, enabling a single model to perform multiple tasks, thereby optimizing computational resources.

Scalability and Efficiency

Ludwig is built for scale. It supports distributed training and offers features like automatic batch size selection, making it easier to train large models efficiently. With the integration of technologies like DeepSpeed, Ludwig ensures that you can train models that are too large for a single GPU or even a single node.

Expert-Level Control

For those who wish to dive deeper, Ludwig provides expert-level control over the models. You can customize everything from activation functions to optimization algorithms. It also supports hyperparameter optimization for fine-tuning model performance.


Ludwig is not just a tool for building models; it's also engineered for production. It offers pre-built Docker containers and native support for running models on Kubernetes. You can also export models to various formats like Torchscript and Triton for easy deployment.

Integration with Deepspeed

Ludwig now integrates with Deepspeed, enabling data and model parallel training. This allows for the training of models that are too large to fit into a single GPU or even a single node, thus making Ludwig more scalable and efficient.

Parameter Efficient Fine-Tuning (PEFT)

PEFT techniques like Low-rank adaptation (LoRA) are now natively supported in Ludwig 0.8. These techniques reduce the number of trainable parameters, speeding up the fine-tuning process and making it more resource-efficient.

Quantized Training (QLoRA)

With the introduction of 4-bit and 8-bit quantized training, Ludwig 0.8 allows for the fine-tuning of large language models on single GPUs. This is particularly useful for those who do not have access to large-scale computing resources.

Prompt Templating

Ludwig 0.8 introduces the ability to use prompt templates for large language models. This feature allows users to provide additional context or instructions to the model, making it more versatile in handling a variety of tasks.

Zero-Shot and In-Context Learning

The new version also supports zero-shot and in-context learning, enabling the model to generalize to tasks it has not been explicitly trained for. This is particularly useful for tasks where labeled data is scarce.

Use-Cases of Ludwig 0.8

Text-Based Applications

  • Chatbots: With the new LLM model type and prompt templating, creating conversational agents is easier than ever.
  • Code Assistants: The fine-tuning capabilities can be leveraged to create intelligent code completion tools.

Data Science and Analytics

  • Automated Data Analysis: The integration with Deepspeed allows for faster processing of large datasets.
  • Predictive Modeling: Parameter Efficient Fine-Tuning (PEFT) enables quick model prototyping for predictive analytics.

Resource-Constrained Environments

  • The 4-bit and 8-bit quantized training options make it feasible to deploy models in resource-constrained environments like IoT devices for Edge computing.

Research and Academia

  • The modular and extensible nature of Ludwig makes it a good fit for academic research where quick experimentation is often required.

What’s Coming with Ludwig 0.9

As Ludwig continues to evolve, the upcoming version 0.9 promises to bring even more features and improvements to the table. Here’s a sneak peek into what’s in store:

Planned Features and Improvements

  • Retrieval Augmented In-Context Learning (RAG): This feature aims to enhance the model’s understanding by dynamically retrieving and inserting contextually relevant information into the prompt. This is particularly useful for tasks that require a deep understanding of the context.
  • Reinforcement Learning from Human Feedback (RLHF): Ludwig 0.9 plans to introduce RLHF, a feature that will allow the model to learn from human feedback, thereby improving its performance on tasks that are difficult to define explicitly.
  • Support for PyTorch 2.0 and Pandas 2.0: With the tech landscape constantly evolving, Ludwig aims to stay up-to-date by offering support for the latest versions of PyTorch and Pandas, ensuring compatibility and performance improvements.



A user interested in utilizing Ludwig 0.8 for building Language Learning Models (LLMs) should first ensure that the following prerequisites are met:

  • Python 3.7 or higher: The programming language in which Ludwig is built.
  • TensorFlow 2.6 or higher: A machine learning framework that Ludwig relies on for various tasks.
  • PyTorch 1.9 or higher: Another machine learning framework that Ludwig can utilize.

These libraries are well-known in the machine learning and deep learning communities and can be easily installed using package managers like pip or conda.

Once the user has a basic grasp of the prerequisites, they can proceed to build their own LLMs using Ludwig 0.8. This tool offers a range of features that make it easier to develop models tailored to specific needs, from data privacy to task-specific performance optimization.

Install Package and Dependencies

For a user interested in building their first Language Learning Model (LLM) using Ludwig 0.8, the following steps can serve as a guide:

First, install Ludwig using pip with the following command:

# !pip uninstall -y tensorflow --quiet
# !pip install ludwig
# !pip install ludwig[llm]

Any existing TensorFlow might affect the package, hence TensorFlow is first installed, and then reinstalled automatically when Ludwig is installed.

Text Wrapping

Enable text wrapping so that you don't have to scroll horizontally and create a function to flush CUDA cache.

from IPython.display import HTML, display

def set_css():

get_ipython().events.register('pre_run_cell', set_css)

def clear_cache():
  if torch.cuda.is_available():
    model = None

Set Up Hugging Face Token 

Hugging Face token is required through the work and hence we are going to have to run Ludwig. Llama 2 model is also required, as it is not openly-accessible and requires requesting for access. Hence, obtain a HuggingFace API Token and request access to Llama2-7b-hf before proceeding.

import getpass
import locale; locale.getpreferredencoding = lambda: "UTF-8"
import logging
import os
import torch
import yaml

from ludwig.api import LudwigModel
os.environ["HUGGING_FACE_HUB_TOKEN"] = getpass.getpass("Token:")
assert os.environ["HUGGING_FACE_HUB_TOKEN"]

The code will request for the token, which must be typed.

Import the Code Generation Dataset

from google.colab import data_table; data_table.enable_dataframe_formatter()
import numpy as np; np.random.seed(123)
import pandas as pd
df = pd.read_json("")
total_rows = len(df)
split_0_count = int(total_rows * 0.9)
split_1_count = int(total_rows * 0.05)
split_2_count = total_rows - split_0_count - split_1_count

# Create an array with split values based on the counts
split_values = np.concatenate([
    np.full(split_2_count, 2)])
# Shuffle the array to ensure randomness
# Add the 'split' column to the DataFrame
df['split'] = split_values
df['split'] = df['split'].astype(int)

The dataset is pretty balanced in terms of the number of examples of each type of instruction (also true for the full dataset with 20,000 rows).

num_self_sufficient = (df['input'] == '').sum()
num_need_contex = df.shape[0] - num_self_sufficient
# We are only using 100 rows of this dataset for this webinar
print(f"Total number of examples in the dataset: {df.shape[0]}")
print(f"% of examples that are self-sufficient: {round(num_self_sufficient/df.shape[0] * 100, 2)}")
print(f"% of examples that are need additional context: {round(num_need_contex/df.shape[0] * 100, 2)}")

Another important consideration is the average character count in the dataset's three columns: instruction, input, and output. Generally, one token corresponds to every 3-4 characters, and there's a token limit imposed by large language models for input processing.

For the base LLaMA-2 model, the maximum context length is capped at 4096 tokens. Ludwig takes care of texts that exceed this limit by automatically truncating them. However, given the typical sequence lengths in our dataset, it appears that we can fine-tune the model using complete examples without the need for truncation.

# Calculating the length of each cell in each column
df['num_characters_instruction'] = df['instruction'].apply(lambda x: len(x))
df['num_characters_input'] = df['input'].apply(lambda x: len(x))
df['num_characters_output'] = df['output'].apply(lambda x: len(x))

# Show Distribution
df.hist(column=['num_characters_instruction', 'num_characters_input', 'num_characters_output'])

# Calculating the average
average_chars_instruction = df['num_characters_instruction'].mean()
average_chars_input = df['num_characters_input'].mean()
average_chars_output = df['num_characters_output'].mean()

print(f'Average number of tokens in the instruction column: {(average_chars_instruction / 3):.0f}')
print(f'Average number of tokens in the input column: {(average_chars_input / 3):.0f}')
print(f'Average number of tokens in the output column: {(average_chars_output / 3):.0f}', end="\n\n")

Average number of tokens in the instruction column: 23 

Average number of tokens in the input column: 8 

Average number of tokens in the output column: 65


Once this is done, when a prompt is given, an output would be shown. 

Additional Tips:

  • Start Simple: Begin with a straightforward dataset and task to familiarize yourself with Ludwig’s functionalities and to troubleshoot any issues that may arise.
  • Consult Documentation: Ludwig’s documentation is comprehensive and can be a valuable resource for understanding its features. The documentation is available at Ludwig’s official website.
  • Experiment: Ludwig offers a wide range of configuration settings and training parameters. Don’t hesitate to experiment with these to find the optimal settings for your specific dataset and task.

By following these steps and tips, a user can build their first LLM using Ludwig 0.8, benefiting from its ease of use and versatility.


The journey from Ludwig 0.7 to 0.8 has been one of significant evolution, marked by the introduction of a range of features that have made the framework more powerful, scalable, and user-friendly. From the integration of Deepspeed for efficient training to the introduction of Parameter Efficient Fine-Tuning (PEFT), Ludwig 0.8 has addressed many of the limitations of its predecessor. The addition of features like Quantized Training (QLoRA) and Prompt Templating further cements its position as a versatile tool for building custom AI models.

Looking ahead, Ludwig 0.9 promises to continue this trajectory of innovation and improvement. With planned features like Retrieval Augmented In-Context Learning (RAG) and Reinforcement Learning from Human Feedback (RLHF), the future of Ludwig looks brighter than ever. The framework's commitment to staying up-to-date with the latest technologies, as evidenced by its planned support for PyTorch 2.0 and Pandas 2.0, ensures that it will remain a relevant and powerful tool in the ever-changing landscape of AI and machine learning.

In summary, Ludwig has proven itself to be more than just another tool in the AI ecosystem. Its low-code, highly customizable nature makes it accessible for both novice and expert users. Whether you're looking to fine-tune large language models, build multi-modal AI systems, or simply experiment with state-of-the-art machine learning techniques, Ludwig offers a robust and flexible framework to meet your needs.

If you need to run Ludwig 0.8, E2E cloud has a large selection of GPUs to select from. NVIDIA H100 is a good fit, as it is highly compatible for LLMs.

Latest Blogs
This is a decorative image for: A Complete Guide To Customer Acquisition For Startups
October 18, 2022

A Complete Guide To Customer Acquisition For Startups

Any business is enlivened by its customers. Therefore, a strategy to constantly bring in new clients is an ongoing requirement. In this regard, having a proper customer acquisition strategy can be of great importance.

So, if you are just starting your business, or planning to expand it, read on to learn more about this concept.

The problem with customer acquisition

As an organization, when working in a diverse and competitive market like India, you need to have a well-defined customer acquisition strategy to attain success. However, this is where most startups struggle. Now, you may have a great product or service, but if you are not in the right place targeting the right demographic, you are not likely to get the results you want.

To resolve this, typically, companies invest, but if that is not channelized properly, it will be futile.

So, the best way out of this dilemma is to have a clear customer acquisition strategy in place.

How can you create the ideal customer acquisition strategy for your business?

  • Define what your goals are

You need to define your goals so that you can meet the revenue expectations you have for the current fiscal year. You need to find a value for the metrics –

  • MRR – Monthly recurring revenue, which tells you all the income that can be generated from all your income channels.
  • CLV – Customer lifetime value tells you how much a customer is willing to spend on your business during your mutual relationship duration.  
  • CAC – Customer acquisition costs, which tells how much your organization needs to spend to acquire customers constantly.
  • Churn rate – It tells you the rate at which customers stop doing business.

All these metrics tell you how well you will be able to grow your business and revenue.

  • Identify your ideal customers

You need to understand who your current customers are and who your target customers are. Once you are aware of your customer base, you can focus your energies in that direction and get the maximum sale of your products or services. You can also understand what your customers require through various analytics and markers and address them to leverage your products/services towards them.

  • Choose your channels for customer acquisition

How will you acquire customers who will eventually tell at what scale and at what rate you need to expand your business? You could market and sell your products on social media channels like Instagram, Facebook and YouTube, or invest in paid marketing like Google Ads. You need to develop a unique strategy for each of these channels. 

  • Communicate with your customers

If you know exactly what your customers have in mind, then you will be able to develop your customer strategy with a clear perspective in mind. You can do it through surveys or customer opinion forms, email contact forms, blog posts and social media posts. After that, you just need to measure the analytics, clearly understand the insights, and improve your strategy accordingly.

Combining these strategies with your long-term business plan will bring results. However, there will be challenges on the way, where you need to adapt as per the requirements to make the most of it. At the same time, introducing new technologies like AI and ML can also solve such issues easily. To learn more about the use of AI and ML and how they are transforming businesses, keep referring to the blog section of E2E Networks.

Reference Links

This is a decorative image for: Constructing 3D objects through Deep Learning
October 18, 2022

Image-based 3D Object Reconstruction State-of-the-Art and trends in the Deep Learning Era

3D reconstruction is one of the most complex issues of deep learning systems. There have been multiple types of research in this field, and almost everything has been tried on it — computer vision, computer graphics and machine learning, but to no avail. However, that has resulted in CNN or convolutional neural networks foraying into this field, which has yielded some success.

The Main Objective of the 3D Object Reconstruction

Developing this deep learning technology aims to infer the shape of 3D objects from 2D images. So, to conduct the experiment, you need the following:

  • Highly calibrated cameras that take a photograph of the image from various angles.
  • Large training datasets can predict the geometry of the object whose 3D image reconstruction needs to be done. These datasets can be collected from a database of images, or they can be collected and sampled from a video.

By using the apparatus and datasets, you will be able to proceed with the 3D reconstruction from 2D datasets.

State-of-the-art Technology Used by the Datasets for the Reconstruction of 3D Objects

The technology used for this purpose needs to stick to the following parameters:

  • Input

Training with the help of one or multiple RGB images, where the segmentation of the 3D ground truth needs to be done. It could be one image, multiple images or even a video stream.

The testing will also be done on the same parameters, which will also help to create a uniform, cluttered background, or both.

  • Output

The volumetric output will be done in both high and low resolution, and the surface output will be generated through parameterisation, template deformation and point cloud. Moreover, the direct and intermediate outputs will be calculated this way.

  • Network architecture used

The architecture used in training is 3D-VAE-GAN, which has an encoder and a decoder, with TL-Net and conditional GAN. At the same time, the testing architecture is 3D-VAE, which has an encoder and a decoder.

  • Training used

The degree of supervision used in 2D vs 3D supervision, weak supervision along with loss functions have to be included in this system. The training procedure is adversarial training with joint 2D and 3D embeddings. Also, the network architecture is extremely important for the speed and processing quality of the output images.

  • Practical applications and use cases

Volumetric representations and surface representations can do the reconstruction. Powerful computer systems need to be used for reconstruction.

Given below are some of the places where 3D Object Reconstruction Deep Learning Systems are used:

  • 3D reconstruction technology can be used in the Police Department for drawing the faces of criminals whose images have been procured from a crime site where their faces are not completely revealed.
  • It can be used for re-modelling ruins at ancient architectural sites. The rubble or the debris stubs of structures can be used to recreate the entire building structure and get an idea of how it looked in the past.
  • They can be used in plastic surgery where the organs, face, limbs or any other portion of the body has been damaged and needs to be rebuilt.
  • It can be used in airport security, where concealed shapes can be used for guessing whether a person is armed or is carrying explosives or not.
  • It can also help in completing DNA sequences.

So, if you are planning to implement this technology, then you can rent the required infrastructure from E2E Networks and avoid investing in it. And if you plan to learn more about such topics, then keep a tab on the blog section of the website

Reference Links

This is a decorative image for: Comprehensive Guide to Deep Q-Learning for Data Science Enthusiasts
October 18, 2022

A Comprehensive Guide To Deep Q-Learning For Data Science Enthusiasts

For all data science enthusiasts who would love to dig deep, we have composed a write-up about Q-Learning specifically for you all. Deep Q-Learning and Reinforcement learning (RL) are extremely popular these days. These two data science methodologies use Python libraries like TensorFlow 2 and openAI’s Gym environment.

So, read on to know more.

What is Deep Q-Learning?

Deep Q-Learning utilizes the principles of Q-learning, but instead of using the Q-table, it uses the neural network. The algorithm of deep Q-Learning uses the states as input and the optimal Q-value of every action possible as the output. The agent gathers and stores all the previous experiences in the memory of the trained tuple in the following order:

State> Next state> Action> Reward

The neural network training stability increases using a random batch of previous data by using the experience replay. Experience replay also means the previous experiences stocking, and the target network uses it for training and calculation of the Q-network and the predicted Q-Value. This neural network uses openAI Gym, which is provided by taxi-v3 environments.

Now, any understanding of Deep Q-Learning   is incomplete without talking about Reinforcement Learning.

What is Reinforcement Learning?

Reinforcement is a subsection of ML. This part of ML is related to the action in which an environmental agent participates in a reward-based system and uses Reinforcement Learning to maximize the rewards. Reinforcement Learning is a different technique from unsupervised learning or supervised learning because it does not require a supervised input/output pair. The number of corrections is also less, so it is a highly efficient technique.

Now, the understanding of reinforcement learning is incomplete without knowing about Markov Decision Process (MDP). MDP is involved with each state that has been presented in the results of the environment, derived from the state previously there. The information which composes both states is gathered and transferred to the decision process. The task of the chosen agent is to maximize the awards. The MDP optimizes the actions and helps construct the optimal policy.

For developing the MDP, you need to follow the Q-Learning Algorithm, which is an extremely important part of data science and machine learning.

What is Q-Learning Algorithm?

The process of Q-Learning is important for understanding the data from scratch. It involves defining the parameters, choosing the actions from the current state and also choosing the actions from the previous state and then developing a Q-table for maximizing the results or output rewards.

The 4 steps that are involved in Q-Learning:

  1. Initializing parameters – The RL (reinforcement learning) model learns the set of actions that the agent requires in the state, environment and time.
  2. Identifying current state – The model stores the prior records for optimal action definition for maximizing the results. For acting in the present state, the state needs to be identified and perform an action combination for it.
  3. Choosing the optimal action set and gaining the relevant experience – A Q-table is generated from the data with a set of specific states and actions, and the weight of this data is calculated for updating the Q-Table to the following step.
  4. Updating Q-table rewards and next state determination – After the relevant experience is gained and agents start getting environmental records. The reward amplitude helps to present the subsequent step.  

In case the Q-table size is huge, then the generation of the model is a time-consuming process. This situation requires Deep Q-learning.

Hopefully, this write-up has provided an outline of Deep Q-Learning and its related concepts. If you wish to learn more about such topics, then keep a tab on the blog section of the E2E Networks website.

Reference Links

This is a decorative image for: GAUDI: A Neural Architect for Immersive 3D Scene Generation
October 13, 2022

GAUDI: A Neural Architect for Immersive 3D Scene Generation

The evolution of artificial intelligence in the past decade has been staggering, and now the focus is shifting towards AI and ML systems to understand and generate 3D spaces. As a result, there has been extensive research on manipulating 3D generative models. In this regard, Apple’s AI and ML scientists have developed GAUDI, a method specifically for this job.

An introduction to GAUDI

The GAUDI 3D immersive technique founders named it after the famous architect Antoni Gaudi. This AI model takes the help of a camera pose decoder, which enables it to guess the possible camera angles of a scene. Hence, the decoder then makes it possible to predict the 3D canvas from almost every angle.

What does GAUDI do?

GAUDI can perform multiple functions –

  • The extensions of these generative models have a tremendous effect on ML and computer vision. Pragmatically, such models are highly useful. They are applied in model-based reinforcement learning and planning world models, SLAM is s, or 3D content creation.
  • Generative modelling for 3D objects has been used for generating scenes using graf, pigan, and gsn, which incorporate a GAN (Generative Adversarial Network). The generator codes radiance fields exclusively. Using the 3D space in the scene along with the camera pose generates the 3D image from that point. This point has a density scalar and RGB value for that specific point in 3D space. This can be done from a 2D camera view. It does this by imposing 3D datasets on those 2D shots. It isolates various objects and scenes and combines them to render a new scene altogether.
  • GAUDI also removes GANs pathologies like mode collapse and improved GAN.
  • GAUDI also uses this to train data on a canonical coordinate system. You can compare it by looking at the trajectory of the scenes.

How is GAUDI applied to the content?

The steps of application for GAUDI have been given below:

  • Each trajectory is created, which consists of a sequence of posed images (These images are from a 3D scene) encoded into a latent representation. This representation which has a radiance field or what we refer to as the 3D scene and the camera path is created in a disentangled way. The results are interpreted as free parameters. The problem is optimized by and formulation of a reconstruction objective.
  • This simple training process is then scaled to trajectories, thousands of them creating a large number of views. The model samples the radiance fields totally from the previous distribution that the model has learned.
  • The scenes are thus synthesized by interpolation within the hidden space.
  • The scaling of 3D scenes generates many scenes that contain thousands of images. During training, there is no issue related to canonical orientation or mode collapse.
  • A novel de-noising optimization technique is used to find hidden representations that collaborate in modelling the camera poses and the radiance field to create multiple datasets with state-of-the-art performance in generating 3D scenes by building a setup that uses images and text.

To conclude, GAUDI has more capabilities and can also be used for sampling various images and video datasets. Furthermore, this will make a foray into AR (augmented reality) and VR (virtual reality). With GAUDI in hand, the sky is only the limit in the field of media creation. So, if you enjoy reading about the latest development in the field of AI and ML, then keep a tab on the blog section of the E2E Networks website.

Reference Links

Build on the most powerful infrastructure cloud

A vector illustration of a tech city using latest cloud technologies & infrastructure