Precise protein-ligand binding is essential for various biological processes, and even minor mutations at binding sites can disrupt these interactions, affecting protein function and leading to altered biological outcomes. Understanding how mutations affect protein-ligand interactions is crucial for uncovering disease mechanisms, predicting drug resistance, and guiding drug design. However, accurately predicting mutation effects remains a significant challenge due to limited high-quality data and the complex indirect impacts on ligand binding, as mutations can influence protein stability, function, and interactions with other molecules. Current computational approaches often struggle to generalize across new datasets. DDMuffin addresses this challenge by integrating transfer learning with a graph-based approach. It pre-trains on a comprehensive protein-ligand affinity dataset, then fine-tunes on mutation data to accurately and efficiently predict changes in binding affinity upon mutations. DDMuffin employs graph convolutional networks across three distinct molecular graphs. The residue-level protein graph utilizes ProtT5 protein language model embeddings as node features, with edges defined by residue distances within 5Å, capturing critical spatial-evolutionary relationships between binding pocket residues. The atom-level ligand graph uses atom types and physicochemical properties as node features, with bonds forming the edges. The protein-ligand interaction graph represents nodes with similar atom-level features, with different types of interactions as edges to effectively capture the complex interaction landscape. DDMuffin demonstrated superior predictive performance compared to other methods on both pre-training and fined-tuning tasks, achieving a Pearson’s correlation of up to 0.70. Additionally, the model's node importance scores highlighted key protein residues and essential ligand regions for binding. Providing fast, accurate predictions on both protein-ligand affinity and mutation effects, DDMuffin is a valuable tool for drug discovery, therapeutic design, and resistance research. We believe DDMuffin will aid researchers in investigating the complexities of protein-ligand interactions, offering insights into how mutations impact binding and the stability of complex structures.