I don’t think it is as common as engineers, or artists going between video games and film or television, but it does indeed happen with some regularity. One of my former coworkers actually did both simultaneously - he was a full-time artist at a AAA studio and he designed his own board game (and got it crowdfunded for distribution). Another designer I know of is [Allison Steele], currently a senior designer on World of Warcraft, formerly on Magic: The Gathering. A third designer I know used to do narrative design at a former employer of mine. She went to write tabletop adventure scenarios. A fourth I know of started as a designer, switched to the producer track on games for Microsoft, and is now a producer at a tabletop RPG publisher. It definitely isn’t a lottery-winning-rarity sort of situation.
As for how easy it is to transfer your skills, the answer is that it generally depends on how difficult it is for you to learn new tools. The core principles of game design remain the same - creating the layout for a board game incorporates the same level design principles as creating an ice dungeon, building dice-rolling turn-based mechanics to convey the experience of sailing a ship is system design, crafting a cohesive and succinct story with characters, beats, and agency for the players is narrative design, and so on. However, each designer’s ability to take those concepts and make them work in the new environment is limited by their ability to use the tools and such in order to create those experiences. Tabletop games are much much easier to prototype and test than video games in that regard.
Got a burning question you want answered?