Heat the olive oil in a skillet over medium heat. Add the sliced mushrooms and cook until they are tender and browned, about 5-7 minutes. Season with salt and pepper. Remove from heat and set aside.
In a mixing bowl, combine the ground beef, Worcestershire sauce, garlic powder, salt, and pepper. Mix until just combined, being careful not to overwork the meat. Form the mixture into four equal patties.
Preheat the grill or grill pan to medium-high heat. Grill the burger patties for about 4-5 minutes per side, or until they reach your desired level of doneness.
During the last minute of grilling, place a slice of Swiss cheese on each patty. Cover the grill or pan to melt the cheese.
Toast the burger buns on the grill for about 1-2 minutes, if desired. Place each burger patty on the bottom bun, top with the sautéed mushrooms, and add any additional toppings such as lettuce, tomato, and onion. Spread condiments on the top bun as desired and assemble the burgers.
Serve the Mushroom Swiss Burgers immediately, and enjoy with your favorite sides.