Two PhD Positions on Embodied Foundation Models

Are you interested in performing high-impact artificial intelligence research on embodied foundation models that will enable an autonomous robot to operate in an open world?

Progress in multimodal foundation models has been astonishing in the past few years and allow to equip robots with world knowledge of scenes, objects, and human activities. Robots should then be able to perceive and act upon the sensed world, be it that current solutions require data diversity, task circumstances, and the label vocabulary all to be pre-defined, stationary and controlled. As soon as these ‘closed world’ deep learning assumptions are broken, perceptual understanding suffers and oftentimes catastrophically. Hence, robots equipped with state-of-the-art multimodal perceptual skills will experience great difficulty generalizing to perception tasks in an open world where sensory and semantic conditions will differ considerably from those perceived during training.

Our key research question is: How to enable multimodal perception for robots that is robust to sensory and semantic shifts between training and operation conditions?

What are you going to do?

You will carry out research and development in the areas of embodied foundation models, deep machine learning and computer vision. Topics of interest are test-time generalization, embodied grounding, data scarcity and uncertainly modeling. The research is embedded in the VIS lab group at the University of Amsterdam, and you will actively collaborate within the OpenBots lab, that contains a team of five PhD students, two at the University of Amsterdam (this vacancy) and three at Delft University of Technology (focusing on planning and control). You will work three days a week at the University of Amsterdam and the other two days you will work with all other four PhDs at TNO and the Royal Netherlands Marechaussee where a physical lab environment with several land-robots is available. The project is carried out with supervisors from the Video and Image Sense Lab (Amsterdam) and the Cognitive Robotics Group (Delft). Students at UvA will be supervised by prof. dr. Cees Snoek and dr. ir. Gertjan Burghouts (TNO).

Your tasks will be to:

• Develop new deep learning, computer vision and multimodal learning methods on embodied foundation models.
• Actively collaborate within the OpenBots Lab and contribute to its demonstrators.
• Regularly present internally on your progress.
• Regularly present intermediate research at international conferences and workshops, publish them in proceedings and journals.
• Assist in teaching activities such as lab assistance and student supervision. 
• Complete and defend a PhD thesis within the official appointment duration of four years.

What do you have to offer?

• An MSc degree in Artificial Intelligence, (Applied) Mathematics/Physics, Computer Science, Engineering or related field.
• A strong background/knowledge in machine learning and computer vision, robotics is a plus.
• Solid mathematics foundations, especially statistics, calculus and linear algebra;
• Excellent programming skills, preferably in Python.
• You are highly motivated, independent, and creative.
• Strong communication, presentation and writing skills and excellent command of English.
• Prior publications in relevant machine learning or computer vision conferences or journals are advantageous.

Our offer

A temporary contract for 38 hours per week for the duration of 4 years (the initial contract will be for a period of 18 months and after satisfactory evaluation it will be extended for a total duration of 4 years). The preferred starting date is in 2024. This should lead to a dissertation (PhD thesis). We will draft an educational plan that includes attendance of courses and (international) meetings. We also expect you to assist in teaching undergraduates and master students.

The gross monthly salary, based on 38 hours per week and dependent on relevant experience, ranges between € 2,770 in the first year to € 3,539 in the last year (scale P). UvA additionally offers an extensive package of secondary benefits, including 8% holiday allowance and a year-end bonus of 8.3%. The UFO profile PhD Candidate is applicable. A favourable tax agreement, the ‘30% ruling’, may apply to non-Dutch applicants. The Collective Labour Agreement of Universities of the Netherlands is applicable.

Besides the salary and a vibrant and challenging environment we offer you multiple fringe benefits:

• 232 holiday hours per year (based on fulltime) and extra holidays between Christmas and 1 January;
• Multiple courses to follow from our Teaching and Learning Centre;
• A complete educational program for PhD students;
• Multiple courses on topics such as time management, handling stress and an online learning platform with 100+ different courses;
• 7 weeks birth leave (partner leave) with 100% salary;
• Partly paid parental leave;
• The possibility to set up a workplace at home;
• A pension at ABP for which UvA pays two third part of the contribution;
• The possibility to follow courses to learn Dutch;
• Help with housing for a studio or small apartment when you’re moving from abroad.

Are you curious to read more about our extensive package of secondary employment benefits, take a look here.

About us

The University of Amsterdam (UvA) is the Netherlands' largest university, offering the widest range of academic programmes. At the UvA, 42,000 students, 6,000 staff members and 3,000 PhD candidates study and work in a diverse range of fields, connected by a culture of curiosity.

The Faculty of Science (FNWI) has a student body of around 8,000, as well as 1,800 members of staff working in education, research or support services. Researchers and students at the Faculty of Science are fascinated by every aspect of how the world works, be it elementary particles, the birth of the universe or the functioning of the brain.

The mission of the Informatics Institute (IvI) is to perform curiosity-driven and use-inspired fundamental research in Computer Science. The main research themes are Artificial Intelligence, Computational Science and Systems and Network Engineering. Our research involves complex information systems at large, with a focus on collaborative, data driven, computational and intelligent systems, all with a strong interactive component.

The position is with Prof. dr. Cees Snoek, Professor, head of the Video & Image Sense lab (VIS lab), at the University of Amsterdam. VIS lab is a world-leading lab on Computer Vision and Machine Learning, and has over 40 PhD students, postdoctoral researchers and faculty members working on a broad variety of deep learning, computer vision, and foundation model subjects, like self-supervised learning, diffusion models, and test-time generalization for perception tasks like object detection, instance segmentation and activity recognition. The position is also embedded in the ELLIS Network of Excellence in AI. You will work two days per week with all other four PhDs at TNO and the Royal Netherlands Marechaussee where a physical lab environment with several land-robots is available.

Want to know more about our organisation? Read more about working at the University of Amsterdam.

Any questions?

Do you have any questions or do you require additional information about the position or your future duties? Please contact:

• Prof. dr. Cees Snoek, c.g.m.snoek@uva.nl
• Dr. ir. Gertjan Burghouts, gertjan.burghouts@tno.nl